Jul 24 22:44:30.551966 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 22:44:30.551979 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 22:44:30.551988 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 22:44:30.552296 ip-10-0-136-126 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 22:44:40.666606 ip-10-0-136-126 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 22:44:40.666623 ip-10-0-136-126 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 840a00bd23fb4662aaeff73fd978c68d -- Jul 24 22:46:10.961665 ip-10-0-136-126 systemd[1]: Starting Kubernetes Kubelet... Jul 24 22:46:11.345126 ip-10-0-136-126 kubenswrapper[2571]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:46:11.345126 ip-10-0-136-126 kubenswrapper[2571]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 22:46:11.345126 ip-10-0-136-126 kubenswrapper[2571]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:46:11.345126 ip-10-0-136-126 kubenswrapper[2571]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 22:46:11.345126 ip-10-0-136-126 kubenswrapper[2571]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 22:46:11.345805 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.345177 2571 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 22:46:11.352208 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352186 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.352208 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352204 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.352208 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352210 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.352208 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352213 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.352208 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352215 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352218 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352223 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352227 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352231 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352235 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352238 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352242 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352245 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352247 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352251 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352253 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352256 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352259 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352262 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352264 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352267 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352269 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352272 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.352423 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352274 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352277 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352279 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352282 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352285 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352287 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352290 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352293 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352295 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352298 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352300 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352303 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352305 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352308 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352310 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352314 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352316 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352319 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352321 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352324 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.352882 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352326 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352328 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352331 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352333 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352336 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352339 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352342 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352344 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352347 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352349 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352352 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352354 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352357 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352359 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352361 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352365 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352369 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352372 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352375 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.353356 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352378 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352381 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352383 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352386 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352389 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352404 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352407 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352409 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352413 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352416 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352419 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352422 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352426 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352428 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352431 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352434 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352437 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352439 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352442 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352444 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.353909 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352447 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352450 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352453 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352455 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352458 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352460 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352463 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352466 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352468 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352470 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352473 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352475 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352478 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352481 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352483 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352486 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.352488 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353237 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353244 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353248 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.354446 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353253 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353256 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353259 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353262 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353264 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353267 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353269 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353272 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353274 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353277 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353279 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353283 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353286 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353289 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353293 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353296 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353299 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353301 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353304 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353306 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.354918 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353309 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353311 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353314 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353317 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353319 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353321 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353324 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353326 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353329 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353331 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353334 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353339 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353344 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353346 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353349 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353352 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353355 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353357 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353360 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353363 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.355427 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353366 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353369 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353373 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353376 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353378 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353381 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353383 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353387 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353406 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353409 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353412 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353414 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353417 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353419 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353422 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353424 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353427 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353429 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353432 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.355910 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353434 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353436 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353439 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353441 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353444 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353447 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353449 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353452 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353454 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353459 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353461 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353464 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353467 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353469 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353471 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353474 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353476 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353479 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353481 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353484 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.356354 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353486 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353490 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353492 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353495 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353498 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353500 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353503 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353505 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353508 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353510 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353513 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353515 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353518 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353520 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353523 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353525 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353528 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353916 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353920 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353923 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.356844 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353927 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353929 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353932 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353935 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353937 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353940 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353944 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353948 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353951 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353954 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353957 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353960 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353963 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353966 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353969 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353973 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353975 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353978 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353981 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.357310 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353983 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353986 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353988 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353991 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353993 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353996 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.353999 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354001 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354004 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354006 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354010 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354012 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354015 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354017 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354019 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354022 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354025 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354028 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354030 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354032 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.357803 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354035 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354038 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354040 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354042 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354045 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354047 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354049 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354053 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354055 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354058 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354060 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354063 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354065 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354070 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354072 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354075 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354078 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354080 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354082 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354085 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.358273 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354087 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354090 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354092 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354095 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354099 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354101 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354104 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354107 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354109 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354112 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354114 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354116 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354119 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354121 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354123 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354126 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354128 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354131 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354133 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.358760 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354135 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354141 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354143 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354145 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354148 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354150 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354154 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354156 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354159 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354161 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354164 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354166 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354170 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354174 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354176 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354179 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354181 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354183 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354259 2571 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354269 2571 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354278 2571 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 22:46:11.359249 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354285 2571 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354295 2571 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354302 2571 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354308 2571 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354314 2571 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354319 2571 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354323 2571 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354327 2571 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354330 2571 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354333 2571 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354336 2571 flags.go:64] FLAG: --cgroup-root="" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354339 2571 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354343 2571 flags.go:64] FLAG: --client-ca-file="" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354350 2571 flags.go:64] FLAG: --cloud-provider="external" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354353 2571 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354358 2571 flags.go:64] FLAG: --cluster-domain="" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354360 2571 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354365 2571 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354368 2571 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354372 2571 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354376 2571 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354378 2571 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354381 2571 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 22:46:11.359771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354384 2571 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354388 2571 flags.go:64] FLAG: --contention-profiling="false" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354404 2571 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354407 2571 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354410 2571 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354413 2571 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354417 2571 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354420 2571 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354423 2571 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354426 2571 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354429 2571 flags.go:64] FLAG: --enable-server="true" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354432 2571 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354439 2571 flags.go:64] FLAG: --event-burst="100" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354442 2571 flags.go:64] FLAG: --event-qps="50" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354445 2571 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354448 2571 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354451 2571 flags.go:64] FLAG: --eviction-hard="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354454 2571 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354457 2571 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354460 2571 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354463 2571 flags.go:64] FLAG: --eviction-soft="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354467 2571 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354470 2571 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354474 2571 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354477 2571 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 22:46:11.360305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354480 2571 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354484 2571 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354487 2571 flags.go:64] FLAG: --feature-gates="" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354491 2571 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354493 2571 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354496 2571 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354499 2571 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354502 2571 flags.go:64] FLAG: --healthz-port="10248" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354506 2571 flags.go:64] FLAG: --help="false" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354509 2571 flags.go:64] FLAG: --hostname-override="ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354512 2571 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354514 2571 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354517 2571 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354521 2571 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354524 2571 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354527 2571 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354530 2571 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354532 2571 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354535 2571 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354539 2571 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354542 2571 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354544 2571 flags.go:64] FLAG: --kube-reserved="" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354547 2571 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354550 2571 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 22:46:11.360956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354553 2571 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354556 2571 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354558 2571 flags.go:64] FLAG: --lock-file="" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354561 2571 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354565 2571 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354568 2571 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354574 2571 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354577 2571 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354580 2571 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354584 2571 flags.go:64] FLAG: --logging-format="text" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354587 2571 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354591 2571 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354593 2571 flags.go:64] FLAG: --manifest-url="" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354596 2571 flags.go:64] FLAG: --manifest-url-header="" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354600 2571 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354603 2571 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354607 2571 flags.go:64] FLAG: --max-pods="110" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354610 2571 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354613 2571 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354615 2571 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354618 2571 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354621 2571 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354624 2571 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354627 2571 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354634 2571 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 22:46:11.361536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354637 2571 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354640 2571 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354642 2571 flags.go:64] FLAG: --pod-cidr="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354645 2571 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354651 2571 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354654 2571 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354656 2571 flags.go:64] FLAG: --pods-per-core="0" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354659 2571 flags.go:64] FLAG: --port="10250" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354662 2571 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354665 2571 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0797ee246055f8d30" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354668 2571 flags.go:64] FLAG: --qos-reserved="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354671 2571 flags.go:64] FLAG: --read-only-port="10255" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354675 2571 flags.go:64] FLAG: --register-node="true" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354678 2571 flags.go:64] FLAG: --register-with-taints="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354682 2571 flags.go:64] FLAG: --registry-burst="10" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354685 2571 flags.go:64] FLAG: --registry-qps="5" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354689 2571 flags.go:64] FLAG: --reserved-cpus="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354692 2571 flags.go:64] FLAG: --reserved-memory="" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354696 2571 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354699 2571 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354702 2571 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354705 2571 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354707 2571 flags.go:64] FLAG: --runonce="false" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354710 2571 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354713 2571 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 22:46:11.362119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354716 2571 flags.go:64] FLAG: --seccomp-default="false" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354719 2571 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354721 2571 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354724 2571 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354727 2571 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354730 2571 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354732 2571 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354735 2571 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354738 2571 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354740 2571 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354743 2571 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354746 2571 flags.go:64] FLAG: --system-cgroups="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354749 2571 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354754 2571 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354756 2571 flags.go:64] FLAG: --tls-cert-file="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354759 2571 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354765 2571 flags.go:64] FLAG: --tls-min-version="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354768 2571 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354771 2571 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354775 2571 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354778 2571 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354781 2571 flags.go:64] FLAG: --v="2" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354784 2571 flags.go:64] FLAG: --version="false" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354790 2571 flags.go:64] FLAG: --vmodule="" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354794 2571 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 22:46:11.362773 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.354797 2571 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354912 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354916 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354920 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354923 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354926 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354928 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354933 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354936 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354939 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354942 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354944 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354946 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354949 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354952 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354954 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354957 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354959 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354962 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.363378 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354964 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354968 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354972 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354975 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354977 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354980 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354982 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354987 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354990 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354992 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354994 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.354999 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355002 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355004 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355007 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355009 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355012 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355014 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355017 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355020 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.363835 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355022 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355024 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355027 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355029 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355032 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355035 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355037 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355040 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355042 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355045 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355047 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355050 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355052 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355055 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355057 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355060 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355062 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355064 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355067 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355071 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.364303 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355073 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355076 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355078 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355082 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355084 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355088 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355091 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355094 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355096 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355099 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355101 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355104 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355106 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355109 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355111 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355114 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355116 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355119 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355121 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355123 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.364793 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355126 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355128 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355131 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355141 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355143 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355146 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355148 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355151 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355153 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355155 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355158 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355162 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355165 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355167 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355170 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355175 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355178 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355181 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355191 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.365318 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355194 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.355197 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.355788 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362072 2571 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362089 2571 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362108 2571 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362114 2571 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362172 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362177 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362181 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362185 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362188 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362191 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362194 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.365791 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362197 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362200 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362202 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362205 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362207 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362210 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362213 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362215 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362218 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362220 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362223 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362225 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362228 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362231 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362233 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362236 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362238 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362240 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362243 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362245 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.366121 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362248 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362250 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362254 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362257 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362260 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362262 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362265 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362267 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362270 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362273 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362276 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362278 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362281 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362283 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362285 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362288 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362290 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362293 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362295 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362298 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.366595 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362301 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362303 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362306 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362308 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362311 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362313 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362316 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362319 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362322 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362325 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362327 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362330 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362332 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362335 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362338 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362346 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362349 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362352 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362354 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362357 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.367063 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362359 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362362 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362364 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362367 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362370 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362374 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362377 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362379 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362382 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362385 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362388 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362403 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362406 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362409 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362412 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362415 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362421 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362425 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362430 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.367569 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362434 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362437 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362439 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362441 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362444 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362447 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362449 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362452 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362455 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362464 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362467 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362471 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362474 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362480 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362625 2571 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 22:46:11.368062 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362629 2571 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362641 2571 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362644 2571 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362646 2571 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362649 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362652 2571 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362655 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362657 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362659 2571 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362662 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362664 2571 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362667 2571 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362669 2571 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362672 2571 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362674 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362676 2571 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362680 2571 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362682 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362684 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362687 2571 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 22:46:11.368496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362689 2571 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362692 2571 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362695 2571 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362698 2571 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362702 2571 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362705 2571 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362707 2571 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362716 2571 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362718 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362721 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362724 2571 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362726 2571 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362729 2571 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362731 2571 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362733 2571 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362736 2571 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362739 2571 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362741 2571 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362743 2571 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 22:46:11.368988 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362746 2571 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362748 2571 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362751 2571 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362753 2571 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362755 2571 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362758 2571 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362760 2571 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362763 2571 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362766 2571 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362770 2571 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362774 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362777 2571 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362780 2571 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362782 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362785 2571 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362788 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362791 2571 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362793 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362796 2571 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362799 2571 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 22:46:11.369603 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362801 2571 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362809 2571 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362812 2571 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362814 2571 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362817 2571 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362819 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362822 2571 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362824 2571 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362826 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362829 2571 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362833 2571 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362836 2571 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362838 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362841 2571 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362843 2571 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362846 2571 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362848 2571 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362850 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362853 2571 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 22:46:11.370180 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362855 2571 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362857 2571 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362860 2571 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362862 2571 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362865 2571 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362867 2571 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362870 2571 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362872 2571 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362874 2571 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362877 2571 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362879 2571 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362882 2571 feature_gate.go:328] unrecognized feature gate: Example Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362884 2571 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362887 2571 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362889 2571 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362898 2571 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362901 2571 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362903 2571 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362906 2571 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 22:46:11.370728 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:11.362908 2571 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.362913 2571 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.363048 2571 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.364932 2571 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.366104 2571 server.go:1019] "Starting client certificate rotation" Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.366216 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.366263 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 22:46:11.371454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.368796 2571 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:46:11.373605 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.373583 2571 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 22:46:11.389422 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.389405 2571 log.go:25] "Validated CRI v1 runtime API" Jul 24 22:46:11.391379 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.391363 2571 log.go:25] "Validated CRI v1 image API" Jul 24 22:46:11.392212 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.392197 2571 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 22:46:11.394494 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.394476 2571 fs.go:135] Filesystem UUIDs: map[1f7be370-92de-45af-91f6-d76385b2bd15:/dev/nvme0n1p4 3f3a4161-84a0-49ad-97fa-315f97c4d5f3:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 24 22:46:11.394540 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.394494 2571 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 22:46:11.397984 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.397966 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:46:11.399512 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.399406 2571 manager.go:216] Machine: {Timestamp:2026-07-24 22:46:11.398234831 +0000 UTC m=+0.327574032 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3199891 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2fabd7eae457ec405a26903f43bd9b SystemUUID:ec2fabd7-eae4-57ec-405a-26903f43bd9b BootID:840a00bd-23fb-4662-aaef-f73fd978c68d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3a:8e:23:49:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3a:8e:23:49:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:e2:10:7a:bf:ff:08 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 22:46:11.399512 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.399505 2571 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 22:46:11.399621 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.399585 2571 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 22:46:11.402372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.402336 2571 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 22:46:11.402528 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.402375 2571 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-126.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 22:46:11.403067 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.403057 2571 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 22:46:11.403107 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.403068 2571 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 22:46:11.403107 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.403076 2571 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:46:11.404367 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.404355 2571 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 22:46:11.405105 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.405095 2571 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 22:46:11.407917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.407905 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:46:11.408053 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408043 2571 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 22:46:11.408083 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408077 2571 kubelet.go:486] "Attempting to sync node with API server" Jul 24 22:46:11.408110 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408090 2571 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 22:46:11.408140 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408115 2571 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 22:46:11.408140 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408128 2571 kubelet.go:398] "Adding apiserver pod source" Jul 24 22:46:11.408200 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408140 2571 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 22:46:11.408607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408594 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:46:11.408663 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.408612 2571 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 22:46:11.410819 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.410800 2571 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 22:46:11.412708 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.412692 2571 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 22:46:11.412762 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.412739 2571 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 22:46:11.414062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414049 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414067 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414073 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414079 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414088 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414093 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414104 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414110 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 22:46:11.414114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414117 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 22:46:11.414315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414123 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 22:46:11.414315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414140 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 22:46:11.414315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414163 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 22:46:11.414720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414710 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 22:46:11.414755 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.414721 2571 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 22:46:11.417142 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.417097 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 22:46:11.417228 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.417166 2571 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-126.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 22:46:11.417559 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.417521 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 22:46:11.419219 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.419193 2571 server.go:1268] "Started kubelet" Jul 24 22:46:11.419423 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.419327 2571 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 22:46:11.419491 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.419440 2571 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 22:46:11.419537 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.419447 2571 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 22:46:11.420378 ip-10-0-136-126 systemd[1]: Started Kubernetes Kubelet. Jul 24 22:46:11.421367 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.421350 2571 server.go:310] "Adding debug handlers to kubelet server" Jul 24 22:46:11.421479 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.421408 2571 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 22:46:11.424242 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.423376 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18c55b432e403b5e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-07-24 22:46:11.419167582 +0000 UTC m=+0.348506784,LastTimestamp:2026-07-24 22:46:11.419167582 +0000 UTC m=+0.348506784,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Jul 24 22:46:11.427080 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427063 2571 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 22:46:11.427167 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427080 2571 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 22:46:11.427167 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.427152 2571 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 22:46:11.427260 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.427157 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.427599 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427584 2571 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 22:46:11.427599 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427586 2571 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 22:46:11.427725 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427611 2571 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 22:46:11.427778 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427733 2571 reconstruct.go:97] "Volume reconstruction finished" Jul 24 22:46:11.427778 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427740 2571 reconciler.go:29] "Reconciler: start to sync state" Jul 24 22:46:11.427991 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.427976 2571 factory.go:55] Registering systemd factory Jul 24 22:46:11.428052 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428014 2571 factory.go:223] Registration of the systemd container factory successfully Jul 24 22:46:11.428227 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428216 2571 factory.go:153] Registering CRI-O factory Jul 24 22:46:11.428270 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428230 2571 factory.go:223] Registration of the crio container factory successfully Jul 24 22:46:11.428304 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428277 2571 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 22:46:11.428304 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428299 2571 factory.go:103] Registering Raw factory Jul 24 22:46:11.428367 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428309 2571 manager.go:1196] Started watching for new ooms in manager Jul 24 22:46:11.428670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.428660 2571 manager.go:349] Starting recovery of all containers Jul 24 22:46:11.429045 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.429015 2571 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 22:46:11.429179 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.429157 2571 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-126.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 22:46:11.439144 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.439121 2571 manager.go:354] Recovery completed Jul 24 22:46:11.443054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.443041 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.444934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.444911 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.444984 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.444946 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.444984 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.444960 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.446254 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.446227 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x24td" Jul 24 22:46:11.446630 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.446612 2571 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 22:46:11.446630 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.446630 2571 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 22:46:11.446767 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.446648 2571 state_mem.go:36] "Initialized new in-memory state store" Jul 24 22:46:11.446767 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.446656 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.447744 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.447721 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.447827 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.447758 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.447827 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.447774 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.449513 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.449498 2571 policy_none.go:49] "None policy: Start" Jul 24 22:46:11.449607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.449521 2571 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 22:46:11.449607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.449531 2571 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 22:46:11.451473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.451462 2571 policy_none.go:47] "Start" Jul 24 22:46:11.452574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.452558 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x24td" Jul 24 22:46:11.457301 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.457213 2571 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-126.ec2.internal.18c55b432fc96add default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-126.ec2.internal,UID:ip-10-0-136-126.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-126.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-126.ec2.internal,},FirstTimestamp:2026-07-24 22:46:11.444935389 +0000 UTC m=+0.374274592,LastTimestamp:2026-07-24 22:46:11.444935389 +0000 UTC m=+0.374274592,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-126.ec2.internal,}" Jul 24 22:46:11.476084 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.475948 2571 manager.go:337] "Starting Device Plugin manager" Jul 24 22:46:11.476176 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.476110 2571 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 22:46:11.476176 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476126 2571 server.go:87] "Starting device plugin registration server" Jul 24 22:46:11.476381 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476368 2571 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 22:46:11.476482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476383 2571 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 22:46:11.476589 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476575 2571 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 22:46:11.476658 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476648 2571 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 22:46:11.476707 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.476659 2571 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 22:46:11.477036 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.476993 2571 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 22:46:11.477036 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.477036 2571 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.492751 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.492730 2571 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2608 into the system container \"/system.slice\": " Jul 24 22:46:11.552276 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.552247 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 22:46:11.554215 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.553360 2571 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 22:46:11.554215 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.553413 2571 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 22:46:11.554215 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.553431 2571 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 22:46:11.554215 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.553466 2571 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 22:46:11.555728 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.555708 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:46:11.577502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.577486 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.578456 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.578440 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.578521 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.578470 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.578521 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.578480 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.578521 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.578505 2571 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.587404 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.587380 2571 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.587478 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.587415 2571 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-126.ec2.internal\": node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.611363 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.611290 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.654143 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.654106 2571 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal"] Jul 24 22:46:11.654243 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.654184 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.655190 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.655175 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.655258 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.655204 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.655258 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.655214 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.657505 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.657493 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.657635 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.657601 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.657762 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.657645 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.658229 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658212 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.658297 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658245 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.658297 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658259 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.658297 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658216 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.658414 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658315 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.658414 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.658325 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.660303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.660290 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.660384 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.660312 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:11.660980 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.660966 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:11.661038 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.660991 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:11.661038 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.661005 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:11.690122 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.690096 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.694459 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.694444 2571 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-126.ec2.internal\" not found" node="ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.711501 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.711481 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.730299 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.730279 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.730386 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.730308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.730386 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.730325 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf90ceb74d2562be9fba76df0fbf255c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"bf90ceb74d2562be9fba76df0fbf255c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.811771 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.811736 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.831078 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831055 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.831153 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831082 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.831153 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf90ceb74d2562be9fba76df0fbf255c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"bf90ceb74d2562be9fba76df0fbf255c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.831234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831181 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.831287 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831253 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf90ceb74d2562be9fba76df0fbf255c-config\") pod \"kube-apiserver-proxy-ip-10-0-136-126.ec2.internal\" (UID: \"bf90ceb74d2562be9fba76df0fbf255c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.831325 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.831300 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/17457ce2fa8ddf46113d3222bb973d5a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal\" (UID: \"17457ce2fa8ddf46113d3222bb973d5a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.912554 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:11.912474 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:11.993032 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.992995 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:11.998062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:11.997692 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:12.013030 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.013004 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.113491 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.113464 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.214067 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.213988 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.314479 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.314451 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.366023 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.365988 2571 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 22:46:12.366467 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.366150 2571 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 22:46:12.415464 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.415438 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.457294 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.457249 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 22:41:12 +0000 UTC" deadline="2028-03-06 05:04:03.733713661 +0000 UTC" Jul 24 22:46:12.457294 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.457291 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14166h17m51.276426784s" Jul 24 22:46:12.515617 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.515549 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.562279 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:12.562250 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf90ceb74d2562be9fba76df0fbf255c.slice/crio-5349b956f1d1280dcfb6895d59787569dfcfda9ddca1b64bd74d6b9d1a6f863b WatchSource:0}: Error finding container 5349b956f1d1280dcfb6895d59787569dfcfda9ddca1b64bd74d6b9d1a6f863b: Status 404 returned error can't find the container with id 5349b956f1d1280dcfb6895d59787569dfcfda9ddca1b64bd74d6b9d1a6f863b Jul 24 22:46:12.562787 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:12.562768 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17457ce2fa8ddf46113d3222bb973d5a.slice/crio-53d9e5f55ae74cf758063f27e67b2d9e9244ab05e4be48c434cdb79b4563b123 WatchSource:0}: Error finding container 53d9e5f55ae74cf758063f27e67b2d9e9244ab05e4be48c434cdb79b4563b123: Status 404 returned error can't find the container with id 53d9e5f55ae74cf758063f27e67b2d9e9244ab05e4be48c434cdb79b4563b123 Jul 24 22:46:12.567502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.567488 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:46:12.616572 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.616532 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.630776 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.630759 2571 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 22:46:12.631875 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.631855 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientMemory" Jul 24 22:46:12.631979 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.631885 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 22:46:12.631979 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.631895 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeHasSufficientPID" Jul 24 22:46:12.717664 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.717631 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.813766 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.813736 2571 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:46:12.817830 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:12.817811 2571 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-126.ec2.internal\" not found" Jul 24 22:46:12.826678 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.826660 2571 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:46:12.827720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.827706 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" Jul 24 22:46:12.829155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.829141 2571 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:46:12.844174 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.844155 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:46:12.845381 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.845368 2571 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" Jul 24 22:46:12.869810 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:12.869792 2571 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 22:46:13.354337 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.354305 2571 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 22:46:13.410408 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.410373 2571 apiserver.go:52] "Watching apiserver" Jul 24 22:46:13.415709 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.415684 2571 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 22:46:13.416139 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.416113 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-lzdj7","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal","openshift-multus/multus-additional-cni-plugins-wn7sf","openshift-multus/multus-dzzpw","openshift-ovn-kubernetes/ovnkube-node-85sr5","kube-system/global-pull-secret-syncer-tvlk8","kube-system/konnectivity-agent-hvwbp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd","openshift-image-registry/node-ca-tzrc7","openshift-multus/network-metrics-daemon-867cq","openshift-network-diagnostics/network-check-target-knh98","openshift-network-operator/iptables-alerter-k4ktr","kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal","openshift-cluster-node-tuning-operator/tuned-tnr8q"] Jul 24 22:46:13.418853 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.418834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.421069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.421045 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:13.421180 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.421120 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:13.421573 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.421550 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c457g\"" Jul 24 22:46:13.421678 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.421591 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 22:46:13.421908 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.421883 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 22:46:13.423463 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.423440 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.425811 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.425792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.427718 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.427701 2571 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 22:46:13.428423 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428390 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.428804 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428698 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.428804 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428759 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 22:46:13.428952 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428914 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 22:46:13.429011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428981 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-k8kk5\"" Jul 24 22:46:13.429011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.428994 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 22:46:13.429205 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.429186 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lgl47\"" Jul 24 22:46:13.429303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.429290 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 22:46:13.429393 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.429372 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.430797 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.430777 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.430890 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.430847 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:13.432407 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.432388 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 22:46:13.432892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.432856 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.432985 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.432914 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 22:46:13.432985 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.432919 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s6llp\"" Jul 24 22:46:13.433607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.433587 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.434415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.434395 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 22:46:13.434662 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.434638 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.434769 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.434685 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 22:46:13.435188 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.435170 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.436072 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.436053 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.436326 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.436305 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.436416 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.436378 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xf75t\"" Jul 24 22:46:13.436555 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.436536 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 22:46:13.438511 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.438491 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.438982 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.438961 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.439232 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.439215 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xn59k\"" Jul 24 22:46:13.439918 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.439898 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.440224 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440193 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.440473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440408 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-slash\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440472 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-conf-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440496 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-multus-certs\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440509 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440521 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440545 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-bin\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440561 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pt9d6\"" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440569 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440596 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-config\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440634 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g9st\" (UniqueName: \"kubernetes.io/projected/e411847a-4117-4ccb-8b67-93fb9eca114d-kube-api-access-9g9st\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440645 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb05b4bb-3520-468d-b4c6-2290d6293cf5-konnectivity-ca\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt89d\" (UniqueName: \"kubernetes.io/projected/9262413d-ed89-49ec-953f-dd286332ff8a-kube-api-access-lt89d\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-ovn\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440737 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440754 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e411847a-4117-4ccb-8b67-93fb9eca114d-ovn-node-metrics-cert\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440786 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-script-lib\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440835 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-os-release\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440859 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-systemd-units\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-netns\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440911 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-daemon-config\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440951 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-etc-kubernetes\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.440994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.440987 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-log-socket\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441012 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-netd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441034 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-env-overrides\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441060 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-cnibin\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441102 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-os-release\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-systemd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441177 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb05b4bb-3520-468d-b4c6-2290d6293cf5-agent-certs\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.441174 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441224 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-system-cni-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441296 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-system-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441317 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cnibin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441330 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-multus\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441385 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-kubelet\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441425 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9md\" (UniqueName: \"kubernetes.io/projected/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-kube-api-access-br9md\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.441666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441451 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-kubelet\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441478 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cni-binary-copy\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-bin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441550 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-netns\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441611 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-var-lib-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441648 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-etc-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441684 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-node-log\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441731 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441766 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441802 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-socket-dir-parent\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441830 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-k8s-cni-cncf-io\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.442574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.441858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-hostroot\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.444080 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.444058 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.446798 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.446774 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.447196 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.447172 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.447616 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.447543 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f6mpq\"" Jul 24 22:46:13.448439 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.448212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.450807 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.450788 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-xxz4q\"" Jul 24 22:46:13.450899 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.450832 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:13.450899 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.450879 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:13.451326 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.451214 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 22:46:13.456236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.456216 2571 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 22:46:13.484125 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.484102 2571 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgp9b" Jul 24 22:46:13.498795 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.498774 2571 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jgp9b" Jul 24 22:46:13.528872 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.528852 2571 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 22:46:13.542374 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542349 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-os-release\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.542501 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542394 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae193ef-02ba-4de6-bc10-c521f0ee99e9-tmp-dir\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.542501 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542436 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.542501 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542460 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-sys\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.542501 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542486 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-tuned\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542513 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-daemon-config\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-etc-kubernetes\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542563 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-log-socket\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-env-overrides\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542623 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-tmp\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542635 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-os-release\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542651 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-os-release\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542697 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.542742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542730 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-systemd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb05b4bb-3520-468d-b4c6-2290d6293cf5-agent-certs\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg2c6\" (UniqueName: \"kubernetes.io/projected/dae193ef-02ba-4de6-bc10-c521f0ee99e9-kube-api-access-pg2c6\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542811 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa66ba0-4324-4a3e-8586-35ebbf372792-host-slash\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542853 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-system-cni-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542905 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cnibin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542932 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-kubelet\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.542958 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-etc-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543007 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-etc-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543007 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-kubelet\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543012 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-os-release\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543039 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-system-cni-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543056 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cnibin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543065 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543084 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-systemd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543126 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-etc-kubernetes\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.543169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543158 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-conf\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543191 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cni-binary-copy\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543218 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-var-lib-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543262 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-node-log\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543262 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-daemon-config\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543301 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e411847a-4117-4ccb-8b67-93fb9eca114d-ovn-node-metrics-cert\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543555 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543771 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-env-overrides\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543807 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-var-lib-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.543876 2571 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544167 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-cni-binary-copy\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544267 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-log-socket\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-node-log\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544306 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-kubernetes\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544335 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544360 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-device-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.544438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544411 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-config\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544442 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544469 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-serviceca\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdrbs\" (UniqueName: \"kubernetes.io/projected/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kube-api-access-jdrbs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544538 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-multus-certs\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544550 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-bin\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544617 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-dbus\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544627 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-multus-certs\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544644 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt89d\" (UniqueName: \"kubernetes.io/projected/9262413d-ed89-49ec-953f-dd286332ff8a-kube-api-access-lt89d\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544669 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-script-lib\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544695 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dae193ef-02ba-4de6-bc10-c521f0ee99e9-hosts-file\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-systemd\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544744 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544768 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-systemd-units\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.544792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544795 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daa66ba0-4324-4a3e-8586-35ebbf372792-iptables-alerter-script\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544819 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-host\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544859 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-netns\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544904 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-netd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544929 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-kubelet-config\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544959 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-config\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.544960 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545010 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-systemd-units\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545055 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-bin\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545062 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-595hm\" (UniqueName: \"kubernetes.io/projected/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-kube-api-access-595hm\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545086 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-netns\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545098 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-modprobe-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545123 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-lib-modules\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545125 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-cni-netd\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545184 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbbk\" (UniqueName: \"kubernetes.io/projected/71b7ab19-732e-4abd-a919-07c4cf1e3949-kube-api-access-xwbbk\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545214 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-cnibin\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545253 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-host\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.545625 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545299 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-cnibin\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545318 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-run\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545365 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545430 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-system-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545517 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-multus\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545576 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-system-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545596 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-kubelet\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545608 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e411847a-4117-4ccb-8b67-93fb9eca114d-ovnkube-script-lib\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545616 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-multus\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545678 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-kubelet\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9md\" (UniqueName: \"kubernetes.io/projected/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-kube-api-access-br9md\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545747 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb05b4bb-3520-468d-b4c6-2290d6293cf5-konnectivity-ca\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545832 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-cni-binary-copy\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545895 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5tcq\" (UniqueName: \"kubernetes.io/projected/3c870da1-f7fd-4241-a588-26b354f604fe-kube-api-access-b5tcq\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.545960 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546014 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-bin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.546453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-netns\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546103 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlr89\" (UniqueName: \"kubernetes.io/projected/daa66ba0-4324-4a3e-8586-35ebbf372792-kube-api-access-dlr89\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546169 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-socket-dir-parent\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546219 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-k8s-cni-cncf-io\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546243 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-hostroot\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546248 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb05b4bb-3520-468d-b4c6-2290d6293cf5-konnectivity-ca\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546267 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-slash\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546299 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546322 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-conf-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546346 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546373 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g9st\" (UniqueName: \"kubernetes.io/projected/e411847a-4117-4ccb-8b67-93fb9eca114d-kube-api-access-9g9st\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546400 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysconfig\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546406 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9262413d-ed89-49ec-953f-dd286332ff8a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546424 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-ovn\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546468 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-var-lib-kubelet\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.547312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546495 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546563 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-cni-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546621 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-socket-dir-parent\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546688 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-run-k8s-cni-cncf-io\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546730 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-hostroot\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546768 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-slash\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546805 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-openvswitch\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546841 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-multus-conf-dir\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.546876 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-ovn-kubernetes\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.547022 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9262413d-ed89-49ec-953f-dd286332ff8a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.547083 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-host-var-lib-cni-bin\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.547128 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-run-ovn\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.548236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.547168 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e411847a-4117-4ccb-8b67-93fb9eca114d-host-run-netns\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.549489 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.549259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e411847a-4117-4ccb-8b67-93fb9eca114d-ovn-node-metrics-cert\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.549792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.549754 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb05b4bb-3520-468d-b4c6-2290d6293cf5-agent-certs\") pod \"konnectivity-agent-hvwbp\" (UID: \"fb05b4bb-3520-468d-b4c6-2290d6293cf5\") " pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.559117 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.558896 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:13.559117 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.558922 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:13.559117 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.558936 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:13.559117 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.558999 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:14.058978098 +0000 UTC m=+2.988317308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:13.559933 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.559811 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"17457ce2fa8ddf46113d3222bb973d5a","Type":"ContainerStarted","Data":"53d9e5f55ae74cf758063f27e67b2d9e9244ab05e4be48c434cdb79b4563b123"} Jul 24 22:46:13.561952 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.561928 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g9st\" (UniqueName: \"kubernetes.io/projected/e411847a-4117-4ccb-8b67-93fb9eca114d-kube-api-access-9g9st\") pod \"ovnkube-node-85sr5\" (UID: \"e411847a-4117-4ccb-8b67-93fb9eca114d\") " pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.563096 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.563077 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt89d\" (UniqueName: \"kubernetes.io/projected/9262413d-ed89-49ec-953f-dd286332ff8a-kube-api-access-lt89d\") pod \"multus-additional-cni-plugins-wn7sf\" (UID: \"9262413d-ed89-49ec-953f-dd286332ff8a\") " pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.563455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.563392 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"bf90ceb74d2562be9fba76df0fbf255c","Type":"ContainerStarted","Data":"5349b956f1d1280dcfb6895d59787569dfcfda9ddca1b64bd74d6b9d1a6f863b"} Jul 24 22:46:13.565051 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.564915 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9md\" (UniqueName: \"kubernetes.io/projected/3a27c5bb-ee99-4d93-add6-e0be2ac4200c-kube-api-access-br9md\") pod \"multus-dzzpw\" (UID: \"3a27c5bb-ee99-4d93-add6-e0be2ac4200c\") " pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.647238 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647149 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysconfig\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647238 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-var-lib-kubelet\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647238 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647230 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647256 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae193ef-02ba-4de6-bc10-c521f0ee99e9-tmp-dir\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647282 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-sys\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-tuned\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647338 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-var-lib-kubelet\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647348 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-tmp\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647394 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg2c6\" (UniqueName: \"kubernetes.io/projected/dae193ef-02ba-4de6-bc10-c521f0ee99e9-kube-api-access-pg2c6\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647420 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa66ba0-4324-4a3e-8586-35ebbf372792-host-slash\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647464 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647493 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-conf\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647520 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-kubernetes\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647545 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-device-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647594 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647617 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-serviceca\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647640 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdrbs\" (UniqueName: \"kubernetes.io/projected/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kube-api-access-jdrbs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647683 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-dbus\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dae193ef-02ba-4de6-bc10-c521f0ee99e9-hosts-file\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647733 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-systemd\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647758 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647785 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daa66ba0-4324-4a3e-8586-35ebbf372792-iptables-alerter-script\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647808 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-host\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-kubelet-config\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647852 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-device-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647858 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647883 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-595hm\" (UniqueName: \"kubernetes.io/projected/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-kube-api-access-595hm\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.647917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647901 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-sys\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647908 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-modprobe-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647950 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/daa66ba0-4324-4a3e-8586-35ebbf372792-host-slash\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647963 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-lib-modules\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbbk\" (UniqueName: \"kubernetes.io/projected/71b7ab19-732e-4abd-a919-07c4cf1e3949-kube-api-access-xwbbk\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648025 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-host\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648035 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-kubernetes\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648051 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-run\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648077 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648114 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5tcq\" (UniqueName: \"kubernetes.io/projected/3c870da1-f7fd-4241-a588-26b354f604fe-kube-api-access-b5tcq\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlr89\" (UniqueName: \"kubernetes.io/projected/daa66ba0-4324-4a3e-8586-35ebbf372792-kube-api-access-dlr89\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648147 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-socket-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647998 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647910 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysconfig\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647962 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648236 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-host\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.647997 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-modprobe-d\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.648428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648252 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-etc-selinux\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648259 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae193ef-02ba-4de6-bc10-c521f0ee99e9-tmp-dir\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648321 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-host\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648350 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-run\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648399 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-registration-dir\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648408 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-kubelet-config\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648525 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-lib-modules\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648542 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-dbus\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.648616 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.648678 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:14.148659168 +0000 UTC m=+3.077998375 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648800 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/daa66ba0-4324-4a3e-8586-35ebbf372792-iptables-alerter-script\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648859 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dae193ef-02ba-4de6-bc10-c521f0ee99e9-hosts-file\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648878 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-sys-fs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648929 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-systemd\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.649234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.648955 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-sysctl-conf\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.649964 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.649941 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:13.650684 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:13.650662 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:14.150639903 +0000 UTC m=+3.079979114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:13.650947 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.650925 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-serviceca\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.650947 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.650935 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-tmp\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.651143 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.651120 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/71b7ab19-732e-4abd-a919-07c4cf1e3949-etc-tuned\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.670378 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.670347 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg2c6\" (UniqueName: \"kubernetes.io/projected/dae193ef-02ba-4de6-bc10-c521f0ee99e9-kube-api-access-pg2c6\") pod \"node-resolver-lzdj7\" (UID: \"dae193ef-02ba-4de6-bc10-c521f0ee99e9\") " pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.670658 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.670634 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlr89\" (UniqueName: \"kubernetes.io/projected/daa66ba0-4324-4a3e-8586-35ebbf372792-kube-api-access-dlr89\") pod \"iptables-alerter-k4ktr\" (UID: \"daa66ba0-4324-4a3e-8586-35ebbf372792\") " pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:13.670821 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.670797 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbbk\" (UniqueName: \"kubernetes.io/projected/71b7ab19-732e-4abd-a919-07c4cf1e3949-kube-api-access-xwbbk\") pod \"tuned-tnr8q\" (UID: \"71b7ab19-732e-4abd-a919-07c4cf1e3949\") " pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.671946 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.671922 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-595hm\" (UniqueName: \"kubernetes.io/projected/bf4d6d27-e179-4ea2-b8fb-0172c057f2dd-kube-api-access-595hm\") pod \"node-ca-tzrc7\" (UID: \"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd\") " pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.673436 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.673412 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdrbs\" (UniqueName: \"kubernetes.io/projected/5ca5f5a6-d276-4c43-adb7-424eb0febf3c-kube-api-access-jdrbs\") pod \"aws-ebs-csi-driver-node-s29kd\" (UID: \"5ca5f5a6-d276-4c43-adb7-424eb0febf3c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.673548 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.673453 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5tcq\" (UniqueName: \"kubernetes.io/projected/3c870da1-f7fd-4241-a588-26b354f604fe-kube-api-access-b5tcq\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:13.731796 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.731754 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:13.738656 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.738633 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" Jul 24 22:46:13.751946 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.751917 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-dzzpw" Jul 24 22:46:13.757063 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.757025 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:13.764855 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.764833 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tzrc7" Jul 24 22:46:13.773525 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.773504 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" Jul 24 22:46:13.781563 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.781539 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" Jul 24 22:46:13.789490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.789473 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lzdj7" Jul 24 22:46:13.799518 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:13.799495 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-k4ktr" Jul 24 22:46:14.152529 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.152491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:14.152529 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.152531 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.152556 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152675 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152692 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152711 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152724 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152734 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:15.152719514 +0000 UTC m=+4.082058719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:14.152776 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152676 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:14.153080 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152777 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:15.152759622 +0000 UTC m=+4.082098813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:14.153080 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.152849 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:15.152828046 +0000 UTC m=+4.082167252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:14.502374 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.502274 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:41:14 +0000 UTC" deadline="2028-02-05 05:29:55.29128693 +0000 UTC" Jul 24 22:46:14.502374 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.502323 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13446h43m40.78896794s" Jul 24 22:46:14.554016 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:14.553983 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:14.554183 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:14.554098 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:14.777703 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.777673 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa66ba0_4324_4a3e_8586_35ebbf372792.slice/crio-2d91d708440016a54b7e4d2b85045bca83cd84dba7195982a20d90ce48496538 WatchSource:0}: Error finding container 2d91d708440016a54b7e4d2b85045bca83cd84dba7195982a20d90ce48496538: Status 404 returned error can't find the container with id 2d91d708440016a54b7e4d2b85045bca83cd84dba7195982a20d90ce48496538 Jul 24 22:46:14.780866 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.780838 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb05b4bb_3520_468d_b4c6_2290d6293cf5.slice/crio-f24a462f17a7f1713d6fbeca95ae1c76251c277403e74d5949b0e35d132eac5b WatchSource:0}: Error finding container f24a462f17a7f1713d6fbeca95ae1c76251c277403e74d5949b0e35d132eac5b: Status 404 returned error can't find the container with id f24a462f17a7f1713d6fbeca95ae1c76251c277403e74d5949b0e35d132eac5b Jul 24 22:46:14.785863 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.785842 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca5f5a6_d276_4c43_adb7_424eb0febf3c.slice/crio-2a3c8bba5fbeb9a7c3eb18774b05c12c47e0f91e6058df737f0f84ff9820b350 WatchSource:0}: Error finding container 2a3c8bba5fbeb9a7c3eb18774b05c12c47e0f91e6058df737f0f84ff9820b350: Status 404 returned error can't find the container with id 2a3c8bba5fbeb9a7c3eb18774b05c12c47e0f91e6058df737f0f84ff9820b350 Jul 24 22:46:14.787275 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.787237 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b7ab19_732e_4abd_a919_07c4cf1e3949.slice/crio-42dac81f0b7e1a8af08a3252c763512374fa36cc8bd3b615b2d2993d91e5cc68 WatchSource:0}: Error finding container 42dac81f0b7e1a8af08a3252c763512374fa36cc8bd3b615b2d2993d91e5cc68: Status 404 returned error can't find the container with id 42dac81f0b7e1a8af08a3252c763512374fa36cc8bd3b615b2d2993d91e5cc68 Jul 24 22:46:14.787993 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.787969 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae193ef_02ba_4de6_bc10_c521f0ee99e9.slice/crio-2ac7372b684c26cf8cd978218e8d5d0166b66aad4404c25e92c561a46d5421de WatchSource:0}: Error finding container 2ac7372b684c26cf8cd978218e8d5d0166b66aad4404c25e92c561a46d5421de: Status 404 returned error can't find the container with id 2ac7372b684c26cf8cd978218e8d5d0166b66aad4404c25e92c561a46d5421de Jul 24 22:46:14.789211 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.789186 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a27c5bb_ee99_4d93_add6_e0be2ac4200c.slice/crio-c4693fc0f99e82abe6d3fb4b008fb025c3c99dde8de4ce0850714f5b09eee9b8 WatchSource:0}: Error finding container c4693fc0f99e82abe6d3fb4b008fb025c3c99dde8de4ce0850714f5b09eee9b8: Status 404 returned error can't find the container with id c4693fc0f99e82abe6d3fb4b008fb025c3c99dde8de4ce0850714f5b09eee9b8 Jul 24 22:46:14.790396 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.790356 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf4d6d27_e179_4ea2_b8fb_0172c057f2dd.slice/crio-039eadf631ebfc8fb95e9efa680bc9a64171d8ec2c1293af80a416dba6be4701 WatchSource:0}: Error finding container 039eadf631ebfc8fb95e9efa680bc9a64171d8ec2c1293af80a416dba6be4701: Status 404 returned error can't find the container with id 039eadf631ebfc8fb95e9efa680bc9a64171d8ec2c1293af80a416dba6be4701 Jul 24 22:46:14.791052 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.791020 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode411847a_4117_4ccb_8b67_93fb9eca114d.slice/crio-07452d26d01d79823c28d46f72350420279acee6cf13b107068119e876189552 WatchSource:0}: Error finding container 07452d26d01d79823c28d46f72350420279acee6cf13b107068119e876189552: Status 404 returned error can't find the container with id 07452d26d01d79823c28d46f72350420279acee6cf13b107068119e876189552 Jul 24 22:46:14.792277 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:14.792252 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9262413d_ed89_49ec_953f_dd286332ff8a.slice/crio-e0845c479e470a66f2286b6f75a1b14653a5229df046bf27e612d7d0a5f2a23b WatchSource:0}: Error finding container e0845c479e470a66f2286b6f75a1b14653a5229df046bf27e612d7d0a5f2a23b: Status 404 returned error can't find the container with id e0845c479e470a66f2286b6f75a1b14653a5229df046bf27e612d7d0a5f2a23b Jul 24 22:46:15.160372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.160188 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:15.160372 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160367 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:15.160372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.160382 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160395 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160418 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.160442 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160482 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:17.160463643 +0000 UTC m=+6.089802847 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160498 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160527 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160561 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:17.160548744 +0000 UTC m=+6.089887935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:15.160660 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.160577 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:17.160569719 +0000 UTC m=+6.089908910 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:15.502840 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.502748 2571 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 22:41:14 +0000 UTC" deadline="2028-02-23 22:31:59.574514464 +0000 UTC" Jul 24 22:46:15.502840 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.502795 2571 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13895h45m44.071724925s" Jul 24 22:46:15.557157 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.557123 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:15.557333 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.557260 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:15.557405 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.557343 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:15.557487 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:15.557452 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:15.588112 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.572305 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tzrc7" event={"ID":"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd","Type":"ContainerStarted","Data":"039eadf631ebfc8fb95e9efa680bc9a64171d8ec2c1293af80a416dba6be4701"} Jul 24 22:46:15.593332 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.593300 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lzdj7" event={"ID":"dae193ef-02ba-4de6-bc10-c521f0ee99e9","Type":"ContainerStarted","Data":"2ac7372b684c26cf8cd978218e8d5d0166b66aad4404c25e92c561a46d5421de"} Jul 24 22:46:15.606381 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.606308 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" event={"ID":"71b7ab19-732e-4abd-a919-07c4cf1e3949","Type":"ContainerStarted","Data":"42dac81f0b7e1a8af08a3252c763512374fa36cc8bd3b615b2d2993d91e5cc68"} Jul 24 22:46:15.620490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.618122 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvwbp" event={"ID":"fb05b4bb-3520-468d-b4c6-2290d6293cf5","Type":"ContainerStarted","Data":"f24a462f17a7f1713d6fbeca95ae1c76251c277403e74d5949b0e35d132eac5b"} Jul 24 22:46:15.620645 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.620600 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k4ktr" event={"ID":"daa66ba0-4324-4a3e-8586-35ebbf372792","Type":"ContainerStarted","Data":"2d91d708440016a54b7e4d2b85045bca83cd84dba7195982a20d90ce48496538"} Jul 24 22:46:15.625526 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.625496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerStarted","Data":"e0845c479e470a66f2286b6f75a1b14653a5229df046bf27e612d7d0a5f2a23b"} Jul 24 22:46:15.633508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.630693 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"07452d26d01d79823c28d46f72350420279acee6cf13b107068119e876189552"} Jul 24 22:46:15.637856 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.637813 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzzpw" event={"ID":"3a27c5bb-ee99-4d93-add6-e0be2ac4200c","Type":"ContainerStarted","Data":"c4693fc0f99e82abe6d3fb4b008fb025c3c99dde8de4ce0850714f5b09eee9b8"} Jul 24 22:46:15.643011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.642956 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" event={"ID":"5ca5f5a6-d276-4c43-adb7-424eb0febf3c","Type":"ContainerStarted","Data":"2a3c8bba5fbeb9a7c3eb18774b05c12c47e0f91e6058df737f0f84ff9820b350"} Jul 24 22:46:15.650490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.649770 2571 generic.go:356] "Generic (PLEG): container finished" podID="17457ce2fa8ddf46113d3222bb973d5a" containerID="d1d4486242668a496eba90aef6c8f75e435d4649775a34b613dddc6f3857418d" exitCode=0 Jul 24 22:46:15.650490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.649849 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"17457ce2fa8ddf46113d3222bb973d5a","Type":"ContainerDied","Data":"d1d4486242668a496eba90aef6c8f75e435d4649775a34b613dddc6f3857418d"} Jul 24 22:46:15.657734 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.657374 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" event={"ID":"bf90ceb74d2562be9fba76df0fbf255c","Type":"ContainerStarted","Data":"7c78619c7d5533b8f372add7b9fee7e0aae579ddefd41e1811ab0c784066eed4"} Jul 24 22:46:15.681411 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:15.681209 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-126.ec2.internal" podStartSLOduration=2.681196328 podStartE2EDuration="2.681196328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:15.680942054 +0000 UTC m=+4.610281263" watchObservedRunningTime="2026-07-24 22:46:15.681196328 +0000 UTC m=+4.610535542" Jul 24 22:46:16.553766 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:16.553731 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:16.554255 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:16.553881 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:16.704450 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:16.704409 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" event={"ID":"17457ce2fa8ddf46113d3222bb973d5a","Type":"ContainerStarted","Data":"e0e074ddc5e78b0eb58194810bd5af2bd3406d79ec18ca4a118ea8cbd90ce056"} Jul 24 22:46:17.179197 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:17.179146 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:17.179197 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:17.179197 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:17.179452 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:17.179232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:17.179452 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179385 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:17.179588 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179478 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:21.179458735 +0000 UTC m=+10.108797944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:17.179588 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179570 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:17.179697 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179608 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:21.17959639 +0000 UTC m=+10.108935591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:17.179697 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179691 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:17.179803 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179708 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:17.179803 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179721 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:17.179803 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.179760 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:21.179746505 +0000 UTC m=+10.109085711 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:17.554876 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:17.554218 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:17.554876 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.554355 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:17.554876 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:17.554710 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:17.554876 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:17.554825 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:18.554495 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:18.554458 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:18.554699 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:18.554612 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:19.554819 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:19.554783 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:19.555304 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:19.554945 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:19.555387 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:19.554792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:19.555529 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:19.555499 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:20.554492 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:20.554454 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:20.554697 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:20.554592 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:21.211764 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:21.211813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:21.211847 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.211978 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212038 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.21201877 +0000 UTC m=+18.141357975 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212101 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212136 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.212122879 +0000 UTC m=+18.141462078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212234 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212249 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212261 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:21.212502 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.212295 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:29.212283733 +0000 UTC m=+18.141622940 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:21.555063 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:21.555027 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:21.555307 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.555160 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:21.555567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:21.555546 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:21.555691 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:21.555671 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:22.554493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:22.554455 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:22.555046 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:22.554620 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:23.554256 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:23.554223 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:23.554453 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:23.554352 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:23.554882 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:23.554857 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:23.555277 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:23.554985 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:24.554436 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:24.554388 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:24.554603 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:24.554533 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:25.554375 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:25.554342 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:25.554796 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:25.554342 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:25.554796 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:25.554487 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:25.554796 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:25.554557 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:26.553581 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:26.553549 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:26.553754 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:26.553675 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:27.554410 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:27.554374 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:27.554843 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:27.554515 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:27.554843 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:27.554560 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:27.554843 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:27.554660 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:28.554127 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:28.554096 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:28.554374 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:28.554190 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:29.273622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:29.273591 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:29.273622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:29.273625 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:29.273647 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273779 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273801 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273808 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273814 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273867 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:46:45.273851834 +0000 UTC m=+34.203191043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273883 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:46:45.273877489 +0000 UTC m=+34.203216716 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273887 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:29.273995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.273942 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:46:45.273927583 +0000 UTC m=+34.203266790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:29.554455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:29.554359 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:29.554618 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:29.554365 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:29.554618 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.554499 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:29.554618 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:29.554578 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:30.554137 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:30.554102 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:30.554615 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:30.554219 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:31.554350 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:31.554312 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:31.554819 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:31.554441 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:31.554819 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:31.554495 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:31.554819 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:31.554620 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:32.554053 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:32.554024 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:32.554219 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:32.554143 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:33.554794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.553882 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:33.554794 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:33.554093 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:33.554794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.554590 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:33.554794 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:33.554701 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:33.743569 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.743536 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hvwbp" event={"ID":"fb05b4bb-3520-468d-b4c6-2290d6293cf5","Type":"ContainerStarted","Data":"09e034aae3bde7ed5fc68df1c02971e2f63967a3a14937cdaf26060f098a17af"} Jul 24 22:46:33.745290 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.745265 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="afee38344af2b29d979b43fa0e245543e9b7f54987da2e37b0397071467e08fa" exitCode=0 Jul 24 22:46:33.745445 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.745348 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"afee38344af2b29d979b43fa0e245543e9b7f54987da2e37b0397071467e08fa"} Jul 24 22:46:33.751367 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.748940 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:46:33.751848 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.751821 2571 generic.go:356] "Generic (PLEG): container finished" podID="e411847a-4117-4ccb-8b67-93fb9eca114d" containerID="12f3f63360bd285a7961b1ee9a9ddc64b6c9fe47c83855f4b75546729583c653" exitCode=1 Jul 24 22:46:33.751949 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.751885 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"a65894603dfd866afcbaf67489f4d607791d9c123923f0be82502f414736b6c7"} Jul 24 22:46:33.752013 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.751995 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"a03638125aed7a063cfdd7e500cf114a1cb951e0573f4e003b55df75ba669ccf"} Jul 24 22:46:33.752061 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.752052 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"f7966f5ccc98a86f001ec89f13474f4c5b7f8087d67e98fd81c4d911a6d33b33"} Jul 24 22:46:33.752119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.752098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"d715a2812fd0ad6ac0361a179fe6e714b7e184dd493671db9dd50bc4a32cefcf"} Jul 24 22:46:33.752168 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.752152 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerDied","Data":"12f3f63360bd285a7961b1ee9a9ddc64b6c9fe47c83855f4b75546729583c653"} Jul 24 22:46:33.752220 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.752201 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"2e21a8714d253b2f4f473a22f924ee12074b36c3d45680bb4b202b4a40501ecb"} Jul 24 22:46:33.753540 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.753522 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-dzzpw" event={"ID":"3a27c5bb-ee99-4d93-add6-e0be2ac4200c","Type":"ContainerStarted","Data":"2a64fbde2c0973ef798e72dfe2cd0036f9fc64a48ea504a56f507239a255f189"} Jul 24 22:46:33.755039 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.755019 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" event={"ID":"5ca5f5a6-d276-4c43-adb7-424eb0febf3c","Type":"ContainerStarted","Data":"1ff30b042fff204572da743ec3080cf37183c7cd6765d8190828bf43705e3b9e"} Jul 24 22:46:33.756505 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.756483 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tzrc7" event={"ID":"bf4d6d27-e179-4ea2-b8fb-0172c057f2dd","Type":"ContainerStarted","Data":"0a1d9045f631d555368c26fbb123b840f91e40c06b1b8d759273bbd4ed55ecc4"} Jul 24 22:46:33.757540 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.757421 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hvwbp" podStartSLOduration=3.337371241 podStartE2EDuration="21.757386392s" totalImagesPullingTime="18.420015151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:33.757286483 +0000 UTC m=+22.686625694" watchObservedRunningTime="2026-07-24 22:46:33.757386392 +0000 UTC m=+22.686725604" Jul 24 22:46:33.757864 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.757835 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-126.ec2.internal" podStartSLOduration=20.75782724 podStartE2EDuration="20.75782724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:16.719580026 +0000 UTC m=+5.648919244" watchObservedRunningTime="2026-07-24 22:46:33.75782724 +0000 UTC m=+22.687166451" Jul 24 22:46:33.758184 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.758160 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lzdj7" event={"ID":"dae193ef-02ba-4de6-bc10-c521f0ee99e9","Type":"ContainerStarted","Data":"3d44ddd335517b462a67c1eb7f242a458439dc0072d7ff0d333e7dd4dcf79027"} Jul 24 22:46:33.759618 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.759593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" event={"ID":"71b7ab19-732e-4abd-a919-07c4cf1e3949","Type":"ContainerStarted","Data":"bad7bb4bcc1c08a41d2c93b9c3676949d975db51ef9e3e31de8666b910d02495"} Jul 24 22:46:33.769603 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.769554 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tzrc7" podStartSLOduration=10.937190208 podStartE2EDuration="21.769541345s" totalImagesPullingTime="10.832351137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:33.769430707 +0000 UTC m=+22.698769919" watchObservedRunningTime="2026-07-24 22:46:33.769541345 +0000 UTC m=+22.698880592" Jul 24 22:46:33.806781 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.806732 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-dzzpw" podStartSLOduration=3.067408092 podStartE2EDuration="21.806719129s" totalImagesPullingTime="18.739311037s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:33.805919603 +0000 UTC m=+22.735258870" watchObservedRunningTime="2026-07-24 22:46:33.806719129 +0000 UTC m=+22.736058340" Jul 24 22:46:33.822186 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.822141 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lzdj7" podStartSLOduration=3.408515665 podStartE2EDuration="21.822126338s" totalImagesPullingTime="18.413610673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:33.821813326 +0000 UTC m=+22.751152537" watchObservedRunningTime="2026-07-24 22:46:33.822126338 +0000 UTC m=+22.751465548" Jul 24 22:46:33.837199 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:33.837162 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tnr8q" podStartSLOduration=3.4207944279999998 podStartE2EDuration="21.837148822s" totalImagesPullingTime="18.416354394s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:33.837063455 +0000 UTC m=+22.766402666" watchObservedRunningTime="2026-07-24 22:46:33.837148822 +0000 UTC m=+22.766488029" Jul 24 22:46:34.554288 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:34.554267 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:34.554414 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:34.554359 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:34.639242 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:34.639219 2571 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 22:46:34.762886 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:34.762847 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-k4ktr" event={"ID":"daa66ba0-4324-4a3e-8586-35ebbf372792","Type":"ContainerStarted","Data":"88937b9869f8b2ce78bf60de996262a823622a6fa2a5bebf3581fa8354bb76f5"} Jul 24 22:46:34.764564 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:34.764537 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" event={"ID":"5ca5f5a6-d276-4c43-adb7-424eb0febf3c","Type":"ContainerStarted","Data":"80698c209f94e66d88dbf7991c40d4a30e74a23b3008be87a53b4c00c842c214"} Jul 24 22:46:34.776232 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:34.776164 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-k4ktr" podStartSLOduration=4.360789097 podStartE2EDuration="22.7761499s" totalImagesPullingTime="18.415360803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:34.776150543 +0000 UTC m=+23.705489753" watchObservedRunningTime="2026-07-24 22:46:34.7761499 +0000 UTC m=+23.705489110" Jul 24 22:46:35.489510 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.489357 2571 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T22:46:34.639238019Z","UUID":"14ee4cfa-6ea5-4412-a9d1-abd0201ba0e1","Handler":null,"Name":"","Endpoint":""} Jul 24 22:46:35.490901 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.490877 2571 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 22:46:35.491032 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.490920 2571 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 22:46:35.554159 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.554044 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:35.554159 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.554110 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:35.554494 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:35.554214 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:35.554494 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:35.554302 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:35.599106 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.599069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:35.599684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:35.599664 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:36.554022 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.553853 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:36.554385 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:36.554095 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:36.772588 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.772556 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:46:36.773004 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.772972 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"7a3990de72f5a893f85f9bbb0d9c601ccea82dfcb3c855af1b74b5b2c6b1dbf0"} Jul 24 22:46:36.775154 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.775127 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" event={"ID":"5ca5f5a6-d276-4c43-adb7-424eb0febf3c","Type":"ContainerStarted","Data":"4ce24956014a5770ffc0dc9f11397cf0b23dc94a106e869c1ee2cd6358b907ae"} Jul 24 22:46:36.775282 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.775159 2571 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 24 22:46:36.792151 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:36.792114 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s29kd" podStartSLOduration=4.180179146 podStartE2EDuration="24.792100776s" totalImagesPullingTime="20.61192163s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:36.79155154 +0000 UTC m=+25.720890811" watchObservedRunningTime="2026-07-24 22:46:36.792100776 +0000 UTC m=+25.721439987" Jul 24 22:46:37.216709 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:37.216672 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:37.217382 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:37.217357 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hvwbp" Jul 24 22:46:37.554550 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:37.554524 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:37.554550 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:37.554538 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:37.555235 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:37.554651 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:37.555235 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:37.554784 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:38.554621 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.554593 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:38.555146 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:38.554700 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:38.782086 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.782048 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="77b127342b1e4bc508c07993689ccef9ea70647edbb1e9a813de6fc7d781cf46" exitCode=0 Jul 24 22:46:38.782265 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.782131 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"77b127342b1e4bc508c07993689ccef9ea70647edbb1e9a813de6fc7d781cf46"} Jul 24 22:46:38.785608 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.785585 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:46:38.786017 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.785988 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"5e42c7c917679ab2b1ba994b4d8a99e3fe78975998dbd9df9474d1795440b857"} Jul 24 22:46:38.786340 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.786312 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:38.786445 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.786416 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:38.786495 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.786455 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:38.786495 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.786485 2571 scope.go:117] "RemoveContainer" containerID="12f3f63360bd285a7961b1ee9a9ddc64b6c9fe47c83855f4b75546729583c653" Jul 24 22:46:38.803130 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.803110 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:38.803227 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:38.803222 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:46:39.553852 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:39.553820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:39.554055 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:39.553831 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:39.554055 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:39.553953 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:39.554055 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:39.554022 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:39.791781 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:39.791757 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:46:39.792208 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:39.792117 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" event={"ID":"e411847a-4117-4ccb-8b67-93fb9eca114d","Type":"ContainerStarted","Data":"d0234ad545abc405f254f32df4fde19488dc38166790e47aec9ba68f2cffab64"} Jul 24 22:46:39.817985 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:39.817842 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" podStartSLOduration=9.377405156 podStartE2EDuration="27.81782162s" totalImagesPullingTime="18.440416464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:39.817060692 +0000 UTC m=+28.746399886" watchObservedRunningTime="2026-07-24 22:46:39.81782162 +0000 UTC m=+28.747160832" Jul 24 22:46:40.081975 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.081777 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvlk8"] Jul 24 22:46:40.082144 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.082129 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:40.082277 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:40.082253 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:40.084787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.084759 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knh98"] Jul 24 22:46:40.084875 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.084859 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:40.084963 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:40.084944 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:40.087335 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.087311 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-867cq"] Jul 24 22:46:40.087457 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.087441 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:40.087570 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:40.087550 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:40.796037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.796000 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="bea399611fcf18f63d0619d7cd07a639608b340ebd4b8250f77cf34a6d0db33e" exitCode=0 Jul 24 22:46:40.796386 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:40.796078 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"bea399611fcf18f63d0619d7cd07a639608b340ebd4b8250f77cf34a6d0db33e"} Jul 24 22:46:41.555593 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:41.555560 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:41.555774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:41.555645 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:41.555774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:41.555677 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:41.555774 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:41.555707 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:41.555774 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:41.555756 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:41.555957 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:41.555849 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:42.802782 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:42.802745 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="47d82bb6232828b31f03ec3199ae8f48e1c1799d3c41669155cdbec6c1e579a5" exitCode=0 Jul 24 22:46:42.803528 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:42.802805 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"47d82bb6232828b31f03ec3199ae8f48e1c1799d3c41669155cdbec6c1e579a5"} Jul 24 22:46:43.554351 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:43.554318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:43.554552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:43.554318 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:43.554552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:43.554455 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:43.554552 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:43.554475 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:43.554722 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:43.554572 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:43.554722 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:43.554650 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:45.289861 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.289824 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.289889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.289925 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290027 2571 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290028 2571 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290048 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290074 2571 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290084 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret podName:1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c nodeName:}" failed. No retries permitted until 2026-07-24 22:47:17.290068822 +0000 UTC m=+66.219408031 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret") pod "global-pull-secret-syncer-tvlk8" (UID: "1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c") : object "kube-system"/"original-pull-secret" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290089 2571 projected.go:196] Error preparing data for projected volume kube-api-access-vtklb for pod openshift-network-diagnostics/network-check-target-knh98: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290101 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs podName:3c870da1-f7fd-4241-a588-26b354f604fe nodeName:}" failed. No retries permitted until 2026-07-24 22:47:17.290091939 +0000 UTC m=+66.219431128 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs") pod "network-metrics-daemon-867cq" (UID: "3c870da1-f7fd-4241-a588-26b354f604fe") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 22:46:45.290302 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.290151 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb podName:d89a4944-62c1-4d98-a707-12d2116d3d5d nodeName:}" failed. No retries permitted until 2026-07-24 22:47:17.290136957 +0000 UTC m=+66.219476170 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vtklb" (UniqueName: "kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb") pod "network-check-target-knh98" (UID: "d89a4944-62c1-4d98-a707-12d2116d3d5d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 22:46:45.554768 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.554680 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:45.554768 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.554702 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:45.554967 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:45.554690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:45.554967 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.554804 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-tvlk8" podUID="1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c" Jul 24 22:46:45.554967 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.554899 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-867cq" podUID="3c870da1-f7fd-4241-a588-26b354f604fe" Jul 24 22:46:45.555095 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:45.554986 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knh98" podUID="d89a4944-62c1-4d98-a707-12d2116d3d5d" Jul 24 22:46:46.422153 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.422120 2571 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-126.ec2.internal" event="NodeReady" Jul 24 22:46:46.422563 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.422267 2571 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 22:46:46.453639 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.453606 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:46:46.459002 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.458977 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8"] Jul 24 22:46:46.459157 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.459114 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.460740 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.460717 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 22:46:46.460892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.460870 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 22:46:46.460996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.460893 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59m4s\"" Jul 24 22:46:46.460996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.460976 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 22:46:46.462831 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.462809 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mp6t4"] Jul 24 22:46:46.462946 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.462928 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.465126 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.465105 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.465571 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.465377 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.465571 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.465427 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 22:46:46.465571 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.465387 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ftpfs\"" Jul 24 22:46:46.466505 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.466479 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5"] Jul 24 22:46:46.466648 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.466608 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.466744 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.466717 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 22:46:46.469319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.468968 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 22:46:46.469319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.469174 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 22:46:46.469319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.469225 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.469555 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.469533 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.469947 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.469930 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-95dkb\"" Jul 24 22:46:46.470151 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.470134 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2"] Jul 24 22:46:46.470274 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.470258 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.479606 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.475697 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.479606 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.477288 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.479606 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.477291 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 22:46:46.479606 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.477520 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9xrkj\"" Jul 24 22:46:46.479606 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.478800 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 22:46:46.479926 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.479906 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf"] Jul 24 22:46:46.480057 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.480043 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.481677 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.481656 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 22:46:46.482607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.482584 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.482749 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.482730 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.482818 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.482809 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 22:46:46.483071 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.483053 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 22:46:46.483423 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.483257 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2ncqg\"" Jul 24 22:46:46.484161 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.484005 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mp6t4"] Jul 24 22:46:46.484161 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.484039 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8"] Jul 24 22:46:46.484161 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.484065 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5"] Jul 24 22:46:46.484161 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.484086 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-88d7b5c95-jqkth"] Jul 24 22:46:46.484161 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.484126 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" Jul 24 22:46:46.486089 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.485639 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.486494 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.486474 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7mzhl\"" Jul 24 22:46:46.486702 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.486683 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.487366 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.487348 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ffg2l"] Jul 24 22:46:46.487504 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.487481 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.489254 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489237 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489446 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489536 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489599 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489601 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489604 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k5m62\"" Jul 24 22:46:46.489716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.489709 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.490812 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.490793 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:46:46.491006 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.490832 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd"] Jul 24 22:46:46.491006 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.490896 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.492629 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.492610 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-x7hjr\"" Jul 24 22:46:46.492737 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.492642 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 22:46:46.492737 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.492642 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.494903 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.494880 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv"] Jul 24 22:46:46.494903 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.494901 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.495082 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.495059 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.496079 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.496059 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 22:46:46.499327 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.499307 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s"] Jul 24 22:46:46.499577 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.499558 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 22:46:46.499975 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.499756 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 22:46:46.500088 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.500075 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5flch\"" Jul 24 22:46:46.501031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.500728 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.501031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.500794 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 22:46:46.502939 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.502921 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mk84c"] Jul 24 22:46:46.503309 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.503202 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 22:46:46.503452 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.503434 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-hqqwx\"" Jul 24 22:46:46.503520 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.503467 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 22:46:46.503601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.503583 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.503710 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.503700 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.504039 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.504018 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" Jul 24 22:46:46.506697 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.506673 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wv7b6\"" Jul 24 22:46:46.506800 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.506747 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7cl5g"] Jul 24 22:46:46.507061 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.507042 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.507505 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.507489 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.507894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.507577 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.510422 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.510382 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 22:46:46.510835 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.510814 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 22:46:46.511179 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.511158 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 22:46:46.511935 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.511914 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69gfn\"" Jul 24 22:46:46.514137 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514116 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2"] Jul 24 22:46:46.514216 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514160 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd"] Jul 24 22:46:46.514554 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514536 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf"] Jul 24 22:46:46.514661 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514648 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk84c"] Jul 24 22:46:46.514724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514679 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv"] Jul 24 22:46:46.514724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514700 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ffg2l"] Jul 24 22:46:46.514821 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514726 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-88d7b5c95-jqkth"] Jul 24 22:46:46.514821 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514758 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cl5g"] Jul 24 22:46:46.514911 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.514873 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.516986 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.516877 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l2hpm\"" Jul 24 22:46:46.517295 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.517269 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s"] Jul 24 22:46:46.518162 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.518136 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 22:46:46.518464 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.518223 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 22:46:46.598840 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.598813 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-default-certificate\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.598996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.598864 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c68a4-54dd-4b07-83b4-79cb055eb539-config-volume\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.598996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.598887 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6bsf\" (UniqueName: \"kubernetes.io/projected/5eaa6fba-900c-4fb8-b51c-821f7112fd03-kube-api-access-q6bsf\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.598996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.598913 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.598996 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.598973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m48r\" (UniqueName: \"kubernetes.io/projected/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-kube-api-access-7m48r\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.599152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599007 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.599152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599030 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-service-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.599152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djv75\" (UniqueName: \"kubernetes.io/projected/2e9c68a4-54dd-4b07-83b4-79cb055eb539-kube-api-access-djv75\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.599152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599095 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5d43aaf-5cc3-4545-8815-b91e60c4951f-cert\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.599152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599142 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.599344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599187 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52677\" (UniqueName: \"kubernetes.io/projected/f5d43aaf-5cc3-4545-8815-b91e60c4951f-kube-api-access-52677\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.599344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599219 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.599344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599247 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.599344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599276 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.599344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599311 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5lrn\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599342 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5092f984-2bce-4b55-a5c9-f3e69b726b07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599368 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmnl6\" (UniqueName: \"kubernetes.io/projected/f7feed80-c9f8-4b5b-9f80-bdb116749763-kube-api-access-cmnl6\") pod \"network-check-source-7d88cb55d8-pwg5s\" (UID: \"f7feed80-c9f8-4b5b-9f80-bdb116749763\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599403 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-trusted-ca\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599465 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xfxm\" (UniqueName: \"kubernetes.io/projected/dfcc5b79-9279-4686-a36a-93fc1b62d587-kube-api-access-7xfxm\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599502 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5459q\" (UniqueName: \"kubernetes.io/projected/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-kube-api-access-5459q\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.599542 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcbw\" (UniqueName: \"kubernetes.io/projected/8b4a9c30-aa60-4698-be16-56f1c25a2982-kube-api-access-2xcbw\") pod \"volume-data-source-validator-549c6d54cf-8sjlf\" (UID: \"8b4a9c30-aa60-4698-be16-56f1c25a2982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599558 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599578 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-config\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599605 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-snapshots\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599653 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2kph\" (UniqueName: \"kubernetes.io/projected/5092f984-2bce-4b55-a5c9-f3e69b726b07-kube-api-access-f2kph\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599671 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-stats-auth\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599686 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599705 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.599759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599755 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69cd7378-d549-481c-b80a-5f5f7e140c51-serving-cert\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599809 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfcc5b79-9279-4686-a36a-93fc1b62d587-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfcc5b79-9279-4686-a36a-93fc1b62d587-config\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599880 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599933 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599974 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-tmp\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.599997 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9c68a4-54dd-4b07-83b4-79cb055eb539-tmp-dir\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600047 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ww68\" (UniqueName: \"kubernetes.io/projected/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-kube-api-access-5ww68\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600072 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-serving-cert\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.600156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600125 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.600552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600161 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.600552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600188 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnrg6\" (UniqueName: \"kubernetes.io/projected/69cd7378-d549-481c-b80a-5f5f7e140c51-kube-api-access-hnrg6\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.600552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600228 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d08fc13-992d-4c02-b9a1-a06ac9bec210-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.600552 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.600275 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.700719 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.700634 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.700719 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.700682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.700719 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.700711 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-tmp\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.700740 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700812 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700833 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700836 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.700870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9c68a4-54dd-4b07-83b4-79cb055eb539-tmp-dir\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700908 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.200885868 +0000 UTC m=+36.130225058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700965 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.200947192 +0000 UTC m=+36.130286394 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:46:46.700991 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.700989 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.200983337 +0000 UTC m=+36.130322527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701004 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ww68\" (UniqueName: \"kubernetes.io/projected/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-kube-api-access-5ww68\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-serving-cert\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701040 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701078 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701099 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnrg6\" (UniqueName: \"kubernetes.io/projected/69cd7378-d549-481c-b80a-5f5f7e140c51-kube-api-access-hnrg6\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701125 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d08fc13-992d-4c02-b9a1-a06ac9bec210-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701141 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701141 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-tmp\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701141 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2e9c68a4-54dd-4b07-83b4-79cb055eb539-tmp-dir\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701183 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-default-certificate\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.701230 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:46:46.701353 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.701282 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.201268546 +0000 UTC m=+36.130607757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:46:46.701931 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701528 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c68a4-54dd-4b07-83b4-79cb055eb539-config-volume\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.701931 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6bsf\" (UniqueName: \"kubernetes.io/projected/5eaa6fba-900c-4fb8-b51c-821f7112fd03-kube-api-access-q6bsf\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.701931 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701604 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.701931 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701649 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m48r\" (UniqueName: \"kubernetes.io/projected/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-kube-api-access-7m48r\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.702116 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701981 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.702116 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.701708 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.702116 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702041 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-service-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.702116 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702080 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djv75\" (UniqueName: \"kubernetes.io/projected/2e9c68a4-54dd-4b07-83b4-79cb055eb539-kube-api-access-djv75\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.702116 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702108 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5d43aaf-5cc3-4545-8815-b91e60c4951f-cert\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.702118 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702131 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702155 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52677\" (UniqueName: \"kubernetes.io/projected/f5d43aaf-5cc3-4545-8815-b91e60c4951f-kube-api-access-52677\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.702172 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.202156869 +0000 UTC m=+36.131496074 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702207 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702242 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702308 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2d08fc13-992d-4c02-b9a1-a06ac9bec210-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:46.702344 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702320 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702351 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5lrn\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702379 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5092f984-2bce-4b55-a5c9-f3e69b726b07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702421 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmnl6\" (UniqueName: \"kubernetes.io/projected/f7feed80-c9f8-4b5b-9f80-bdb116749763-kube-api-access-cmnl6\") pod \"network-check-source-7d88cb55d8-pwg5s\" (UID: \"f7feed80-c9f8-4b5b-9f80-bdb116749763\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702448 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-trusted-ca\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702485 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xfxm\" (UniqueName: \"kubernetes.io/projected/dfcc5b79-9279-4686-a36a-93fc1b62d587-kube-api-access-7xfxm\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702510 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5459q\" (UniqueName: \"kubernetes.io/projected/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-kube-api-access-5459q\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702566 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcbw\" (UniqueName: \"kubernetes.io/projected/8b4a9c30-aa60-4698-be16-56f1c25a2982-kube-api-access-2xcbw\") pod \"volume-data-source-validator-549c6d54cf-8sjlf\" (UID: \"8b4a9c30-aa60-4698-be16-56f1c25a2982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702596 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702626 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-config\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.702684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702665 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-snapshots\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702696 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2kph\" (UniqueName: \"kubernetes.io/projected/5092f984-2bce-4b55-a5c9-f3e69b726b07-kube-api-access-f2kph\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702129 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2e9c68a4-54dd-4b07-83b4-79cb055eb539-config-volume\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-stats-auth\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702749 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702788 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702816 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69cd7378-d549-481c-b80a-5f5f7e140c51-serving-cert\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702841 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfcc5b79-9279-4686-a36a-93fc1b62d587-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702849 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-service-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.703069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.702870 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfcc5b79-9279-4686-a36a-93fc1b62d587-config\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.703455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.703290 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/69cd7378-d549-481c-b80a-5f5f7e140c51-snapshots\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.703558 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.703535 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfcc5b79-9279-4686-a36a-93fc1b62d587-config\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.703650 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.703630 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-trusted-ca\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.704198 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.704168 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.704850 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.705000 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69cd7378-d549-481c-b80a-5f5f7e140c51-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.705092 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.705138 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.205123612 +0000 UTC m=+36.134462816 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.705204 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:46.705236 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:47.205225003 +0000 UTC m=+36.134564211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.705440 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-config\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.705961 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.705677 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.706436 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.706639 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5d43aaf-5cc3-4545-8815-b91e60c4951f-cert\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.706685 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-stats-auth\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.706928 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.707005 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-default-certificate\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.707076 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/5092f984-2bce-4b55-a5c9-f3e69b726b07-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.707590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.707291 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-serving-cert\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.708357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.708305 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/69cd7378-d549-481c-b80a-5f5f7e140c51-serving-cert\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.708688 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.708647 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.709004 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.708984 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dfcc5b79-9279-4686-a36a-93fc1b62d587-serving-cert\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.712329 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.711919 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.712329 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.712211 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnrg6\" (UniqueName: \"kubernetes.io/projected/69cd7378-d549-481c-b80a-5f5f7e140c51-kube-api-access-hnrg6\") pod \"insights-operator-5885fdb759-mp6t4\" (UID: \"69cd7378-d549-481c-b80a-5f5f7e140c51\") " pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.713270 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.712751 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m48r\" (UniqueName: \"kubernetes.io/projected/3de1980f-7b67-4fd1-9981-4f952b4ee8b0-kube-api-access-7m48r\") pod \"console-operator-584dcb6b54-ffg2l\" (UID: \"3de1980f-7b67-4fd1-9981-4f952b4ee8b0\") " pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.713357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.713225 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ww68\" (UniqueName: \"kubernetes.io/projected/5ab34ede-a8e4-4faa-8447-c3ae1b89b77c-kube-api-access-5ww68\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2\" (UID: \"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.713964 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.713917 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xfxm\" (UniqueName: \"kubernetes.io/projected/dfcc5b79-9279-4686-a36a-93fc1b62d587-kube-api-access-7xfxm\") pod \"service-ca-operator-5d6c75f84f-mmxlv\" (UID: \"dfcc5b79-9279-4686-a36a-93fc1b62d587\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.714893 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.714870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5lrn\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:46.715152 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.715130 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52677\" (UniqueName: \"kubernetes.io/projected/f5d43aaf-5cc3-4545-8815-b91e60c4951f-kube-api-access-52677\") pod \"ingress-canary-mk84c\" (UID: \"f5d43aaf-5cc3-4545-8815-b91e60c4951f\") " pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:46.715792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.715764 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcbw\" (UniqueName: \"kubernetes.io/projected/8b4a9c30-aa60-4698-be16-56f1c25a2982-kube-api-access-2xcbw\") pod \"volume-data-source-validator-549c6d54cf-8sjlf\" (UID: \"8b4a9c30-aa60-4698-be16-56f1c25a2982\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" Jul 24 22:46:46.716579 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.716371 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2kph\" (UniqueName: \"kubernetes.io/projected/5092f984-2bce-4b55-a5c9-f3e69b726b07-kube-api-access-f2kph\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:46.717423 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.717348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6bsf\" (UniqueName: \"kubernetes.io/projected/5eaa6fba-900c-4fb8-b51c-821f7112fd03-kube-api-access-q6bsf\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:46.719419 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.718314 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djv75\" (UniqueName: \"kubernetes.io/projected/2e9c68a4-54dd-4b07-83b4-79cb055eb539-kube-api-access-djv75\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:46.719497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.719421 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5459q\" (UniqueName: \"kubernetes.io/projected/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-kube-api-access-5459q\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:46.720410 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.720372 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmnl6\" (UniqueName: \"kubernetes.io/projected/f7feed80-c9f8-4b5b-9f80-bdb116749763-kube-api-access-cmnl6\") pod \"network-check-source-7d88cb55d8-pwg5s\" (UID: \"f7feed80-c9f8-4b5b-9f80-bdb116749763\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" Jul 24 22:46:46.796763 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.796742 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" Jul 24 22:46:46.813562 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.813541 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" Jul 24 22:46:46.824727 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.824705 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" Jul 24 22:46:46.839134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.839111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:46.854447 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.854188 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" Jul 24 22:46:46.860798 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.860765 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" Jul 24 22:46:46.869969 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:46.869808 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mk84c" Jul 24 22:46:47.028692 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.028577 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-mp6t4"] Jul 24 22:46:47.033143 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.033075 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2"] Jul 24 22:46:47.046023 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.045961 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf"] Jul 24 22:46:47.075054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.075025 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv"] Jul 24 22:46:47.083309 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.083286 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-ffg2l"] Jul 24 22:46:47.091289 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.091267 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mk84c"] Jul 24 22:46:47.101167 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.101143 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s"] Jul 24 22:46:47.106777 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:47.106710 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b4a9c30_aa60_4698_be16_56f1c25a2982.slice/crio-0cd55a76db17d4cca3b9d651663033c5e0a7a82f7b6bcbe688cfbf72af4ca4b8 WatchSource:0}: Error finding container 0cd55a76db17d4cca3b9d651663033c5e0a7a82f7b6bcbe688cfbf72af4ca4b8: Status 404 returned error can't find the container with id 0cd55a76db17d4cca3b9d651663033c5e0a7a82f7b6bcbe688cfbf72af4ca4b8 Jul 24 22:46:47.107823 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:47.107790 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de1980f_7b67_4fd1_9981_4f952b4ee8b0.slice/crio-56f1aab1dcbf5e32a6907e3edb6508ad89612753825e76c47c9b478f731988ae WatchSource:0}: Error finding container 56f1aab1dcbf5e32a6907e3edb6508ad89612753825e76c47c9b478f731988ae: Status 404 returned error can't find the container with id 56f1aab1dcbf5e32a6907e3edb6508ad89612753825e76c47c9b478f731988ae Jul 24 22:46:47.108257 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:47.108240 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfcc5b79_9279_4686_a36a_93fc1b62d587.slice/crio-d315c9750f054fb9b3b541b6cd89608ff039347d21c58a9974cce4658dda0555 WatchSource:0}: Error finding container d315c9750f054fb9b3b541b6cd89608ff039347d21c58a9974cce4658dda0555: Status 404 returned error can't find the container with id d315c9750f054fb9b3b541b6cd89608ff039347d21c58a9974cce4658dda0555 Jul 24 22:46:47.108944 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:47.108821 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ab34ede_a8e4_4faa_8447_c3ae1b89b77c.slice/crio-3134fa1ab948cb0cd22660c26ef8dc57c23cee6070e4b9dee0335bf2a5c50625 WatchSource:0}: Error finding container 3134fa1ab948cb0cd22660c26ef8dc57c23cee6070e4b9dee0335bf2a5c50625: Status 404 returned error can't find the container with id 3134fa1ab948cb0cd22660c26ef8dc57c23cee6070e4b9dee0335bf2a5c50625 Jul 24 22:46:47.114649 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:46:47.114163 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d43aaf_5cc3_4545_8815_b91e60c4951f.slice/crio-3eb834ca086893fd85e7c0d8314d8fd16b9d9ca58245df9b8a268112c97bd8f0 WatchSource:0}: Error finding container 3eb834ca086893fd85e7c0d8314d8fd16b9d9ca58245df9b8a268112c97bd8f0: Status 404 returned error can't find the container with id 3eb834ca086893fd85e7c0d8314d8fd16b9d9ca58245df9b8a268112c97bd8f0 Jul 24 22:46:47.208046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.207959 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:47.208046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208006 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:47.208046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208119 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208132 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208147 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208128053 +0000 UTC m=+37.137467244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208160 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208176 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208179 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208169197 +0000 UTC m=+37.137508397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208215 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208204823 +0000 UTC m=+37.137544012 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208246 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208253 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208284 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208267662 +0000 UTC m=+37.137606855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208302 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:47.208303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.208305 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:47.208742 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208329 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208318892 +0000 UTC m=+37.137658081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:47.208742 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208347 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208337029 +0000 UTC m=+37.137676219 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:46:47.208742 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208446 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:46:47.208742 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208459 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:46:47.208742 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:47.208514 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:48.208505304 +0000 UTC m=+37.137844496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:46:47.553853 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.553823 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:46:47.554280 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.553827 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:46:47.554280 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.553834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:46:47.556154 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.556139 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:46:47.556340 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.556329 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:46:47.556381 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.556362 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmr4r\"" Jul 24 22:46:47.556529 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.556493 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4wx4\"" Jul 24 22:46:47.818911 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.818639 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" event={"ID":"f7feed80-c9f8-4b5b-9f80-bdb116749763","Type":"ContainerStarted","Data":"5b23d20a847d7a49cb596d2105c0f18e74ff568cd29703d4abf28ad65bf8ceb4"} Jul 24 22:46:47.834536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.833530 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="c297a0dea235632f9c33babf49993a953db62e50acce3e1c405b61950413ffff" exitCode=0 Jul 24 22:46:47.834536 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.833669 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"c297a0dea235632f9c33babf49993a953db62e50acce3e1c405b61950413ffff"} Jul 24 22:46:47.842641 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.841971 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" event={"ID":"8b4a9c30-aa60-4698-be16-56f1c25a2982","Type":"ContainerStarted","Data":"0cd55a76db17d4cca3b9d651663033c5e0a7a82f7b6bcbe688cfbf72af4ca4b8"} Jul 24 22:46:47.849658 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.849567 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" event={"ID":"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c","Type":"ContainerStarted","Data":"3134fa1ab948cb0cd22660c26ef8dc57c23cee6070e4b9dee0335bf2a5c50625"} Jul 24 22:46:47.853459 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.853358 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" event={"ID":"69cd7378-d549-481c-b80a-5f5f7e140c51","Type":"ContainerStarted","Data":"4514729fc6d834b875cf7f18c39f528e064e5619309c196eb8cd6e0e762b417b"} Jul 24 22:46:47.857749 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.857677 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk84c" event={"ID":"f5d43aaf-5cc3-4545-8815-b91e60c4951f","Type":"ContainerStarted","Data":"3eb834ca086893fd85e7c0d8314d8fd16b9d9ca58245df9b8a268112c97bd8f0"} Jul 24 22:46:47.860013 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.859944 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" event={"ID":"dfcc5b79-9279-4686-a36a-93fc1b62d587","Type":"ContainerStarted","Data":"d315c9750f054fb9b3b541b6cd89608ff039347d21c58a9974cce4658dda0555"} Jul 24 22:46:47.865791 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:47.865766 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" event={"ID":"3de1980f-7b67-4fd1-9981-4f952b4ee8b0","Type":"ContainerStarted","Data":"56f1aab1dcbf5e32a6907e3edb6508ad89612753825e76c47c9b478f731988ae"} Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218166 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218202 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218296 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218363 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.218423 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.218549 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.218613 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.218595004 +0000 UTC m=+39.147934209 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219014 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219064 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.219049151 +0000 UTC m=+39.148388355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219123 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219133 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219162 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.219152289 +0000 UTC m=+39.148491484 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219215 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:46:48.219574 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219241 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.21923166 +0000 UTC m=+39.148570862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:46:48.220495 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219301 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.219291654 +0000 UTC m=+39.148630856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:46:48.220495 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219356 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:46:48.220495 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219385 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.219374526 +0000 UTC m=+39.148713729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:46:48.220495 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219452 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:46:48.220495 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:48.219481 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:50.219472224 +0000 UTC m=+39.148811414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:46:48.878014 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.877964 2571 generic.go:356] "Generic (PLEG): container finished" podID="9262413d-ed89-49ec-953f-dd286332ff8a" containerID="9c9266ffc315ded2bfd9d549bd16812c4657eb6ba5954b807fef71d4ebcc844b" exitCode=0 Jul 24 22:46:48.878567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:48.878060 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerDied","Data":"9c9266ffc315ded2bfd9d549bd16812c4657eb6ba5954b807fef71d4ebcc844b"} Jul 24 22:46:49.885627 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:49.885586 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" event={"ID":"9262413d-ed89-49ec-953f-dd286332ff8a","Type":"ContainerStarted","Data":"f92a63ef31a68cfa0bb3a6c1286a7b70afb2f5e7ce70d4d200a9c50d585330f9"} Jul 24 22:46:49.887845 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:49.887804 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" event={"ID":"8b4a9c30-aa60-4698-be16-56f1c25a2982","Type":"ContainerStarted","Data":"b1f4c5aaf29f5b72ae1023bacbe508184c20184f58582802b7ca22081d61a5bb"} Jul 24 22:46:49.907547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:49.907447 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wn7sf" podStartSLOduration=8.769006867 podStartE2EDuration="37.907428258s" totalImagesPullingTime="29.138421391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:49.906784318 +0000 UTC m=+38.836123530" watchObservedRunningTime="2026-07-24 22:46:49.907428258 +0000 UTC m=+38.836767470" Jul 24 22:46:49.926444 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:49.926323 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-8sjlf" podStartSLOduration=32.928744735 podStartE2EDuration="34.926303454s" totalImagesPullingTime="1.997558719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:49.925705405 +0000 UTC m=+38.855044620" watchObservedRunningTime="2026-07-24 22:46:49.926303454 +0000 UTC m=+38.855642666" Jul 24 22:46:50.239455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239358 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:50.239455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239425 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:50.239659 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239530 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:46:50.239659 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239540 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:50.239659 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:50.239659 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239547 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:46:50.239659 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239613 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239531 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239694 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.239674843 +0000 UTC m=+43.169014037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239675 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239673 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239806 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.239746962 +0000 UTC m=+43.169086155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239834 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.239824073 +0000 UTC m=+43.169163269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:50.239879 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239852 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.239842547 +0000 UTC m=+43.169181743 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:46:50.240140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.239889 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.239878861 +0000 UTC m=+43.169218081 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:46:50.240140 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.239952 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:50.240140 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:50.240005 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:50.240140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.240103 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:46:50.240140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.240138 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.240128246 +0000 UTC m=+43.169467435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:46:50.240323 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.240103 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:46:50.240323 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:50.240166 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:46:54.240159324 +0000 UTC m=+43.169498513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:46:54.281060 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281022 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281091 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281113 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281150 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281171 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281183 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281244 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281261 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281242835 +0000 UTC m=+51.210582044 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281278 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281271809 +0000 UTC m=+51.210610998 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281285 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281262 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281252 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281351 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281356 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281339373 +0000 UTC m=+51.210678595 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281432 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281416847 +0000 UTC m=+51.210756036 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:46:54.281561 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:54.281479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:46:54.282041 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281532 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.28151856 +0000 UTC m=+51.210857757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:46:54.282041 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281555 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281547288 +0000 UTC m=+51.210886477 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:46:54.282041 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281577 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:46:54.282041 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:54.281620 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:02.281609611 +0000 UTC m=+51.210948818 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:46:55.906806 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.906778 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/0.log" Jul 24 22:46:55.907357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.907324 2571 generic.go:356] "Generic (PLEG): container finished" podID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" containerID="16d233e3ab61becf7005c97ba2c0b15d0fef1b5d8722693daaeaa2fe5fb03d88" exitCode=255 Jul 24 22:46:55.907655 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.907625 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" event={"ID":"3de1980f-7b67-4fd1-9981-4f952b4ee8b0","Type":"ContainerDied","Data":"16d233e3ab61becf7005c97ba2c0b15d0fef1b5d8722693daaeaa2fe5fb03d88"} Jul 24 22:46:55.907785 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.907714 2571 scope.go:117] "RemoveContainer" containerID="16d233e3ab61becf7005c97ba2c0b15d0fef1b5d8722693daaeaa2fe5fb03d88" Jul 24 22:46:55.909421 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.909187 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" event={"ID":"f7feed80-c9f8-4b5b-9f80-bdb116749763","Type":"ContainerStarted","Data":"b1b43a09fe6f1449ba59015cfd63e726b4fa79da832b5d5c2b5514449b79dc5c"} Jul 24 22:46:55.911366 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.911343 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" event={"ID":"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c","Type":"ContainerStarted","Data":"1dfe950c3979e9b3e190496084516940e222e3a73ea9d4a0d6ff871d597622d1"} Jul 24 22:46:55.913266 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.913247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" event={"ID":"69cd7378-d549-481c-b80a-5f5f7e140c51","Type":"ContainerStarted","Data":"ad9fd25545afa01b01b8ad59ecb0644e4f433aabb63926fff6c61dfcca2961cf"} Jul 24 22:46:55.914776 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.914754 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mk84c" event={"ID":"f5d43aaf-5cc3-4545-8815-b91e60c4951f","Type":"ContainerStarted","Data":"3e7cade5dd10abb07fa6dde9534ca1110688b65628b6968eadcad725d7554c35"} Jul 24 22:46:55.916117 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.916098 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" event={"ID":"dfcc5b79-9279-4686-a36a-93fc1b62d587","Type":"ContainerStarted","Data":"ba296f131ebce1e6c12af85d56ccfc6adba5eb670d558439d84d3d7f9e4898e7"} Jul 24 22:46:55.941922 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.941869 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" podStartSLOduration=33.253179991 podStartE2EDuration="40.941849418s" totalImagesPullingTime="7.688669427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:55.939578089 +0000 UTC m=+44.868917301" watchObservedRunningTime="2026-07-24 22:46:55.941849418 +0000 UTC m=+44.871188634" Jul 24 22:46:55.954222 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.954169 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" podStartSLOduration=33.268449598 podStartE2EDuration="40.954150623s" totalImagesPullingTime="7.685701025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:55.953967522 +0000 UTC m=+44.883306734" watchObservedRunningTime="2026-07-24 22:46:55.954150623 +0000 UTC m=+44.883489836" Jul 24 22:46:55.968190 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.968138 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-pwg5s" podStartSLOduration=33.270092227 podStartE2EDuration="40.968119222s" totalImagesPullingTime="7.698026995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:55.967034927 +0000 UTC m=+44.896374140" watchObservedRunningTime="2026-07-24 22:46:55.968119222 +0000 UTC m=+44.897458435" Jul 24 22:46:55.984733 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:55.984674 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" podStartSLOduration=33.294918227 podStartE2EDuration="40.984654581s" totalImagesPullingTime="7.689736354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:55.983525478 +0000 UTC m=+44.912864689" watchObservedRunningTime="2026-07-24 22:46:55.984654581 +0000 UTC m=+44.913993796" Jul 24 22:46:56.755467 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.755370 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mk84c" podStartSLOduration=3.070348344 podStartE2EDuration="10.755342482s" totalImagesPullingTime="7.684994138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:46:56.008586501 +0000 UTC m=+44.937925712" watchObservedRunningTime="2026-07-24 22:46:56.755342482 +0000 UTC m=+45.684681692" Jul 24 22:46:56.755892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.755869 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c68zt"] Jul 24 22:46:56.777644 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.777609 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c68zt"] Jul 24 22:46:56.777802 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.777735 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.780905 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.780031 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlc68\"" Jul 24 22:46:56.780905 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.780275 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 22:46:56.780905 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.780495 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 22:46:56.812355 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.812319 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.812532 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.812418 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9kzs\" (UniqueName: \"kubernetes.io/projected/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-api-access-f9kzs\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.812532 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.812453 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-data-volume\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.812619 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.812536 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.812676 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.812662 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-crio-socket\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.839250 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.839225 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:56.839370 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.839273 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:46:56.913243 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913203 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-crio-socket\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913255 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913288 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9kzs\" (UniqueName: \"kubernetes.io/projected/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-api-access-f9kzs\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913307 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-data-volume\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913336 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-crio-socket\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:56.913446 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:56.913513 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:57.41349678 +0000 UTC m=+46.342835969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : secret "insights-runtime-extractor-tls" not found Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:56.913600 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:46:56.913699 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:56.913667 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:57.413650465 +0000 UTC m=+46.342989654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : configmap "kube-rbac-proxy" not found Jul 24 22:46:56.914020 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.913733 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-data-volume\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:56.921334 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.921314 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/1.log" Jul 24 22:46:56.921708 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.921695 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/0.log" Jul 24 22:46:56.921794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.921727 2571 generic.go:356] "Generic (PLEG): container finished" podID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" containerID="89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11" exitCode=255 Jul 24 22:46:56.921794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.921765 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" event={"ID":"3de1980f-7b67-4fd1-9981-4f952b4ee8b0","Type":"ContainerDied","Data":"89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11"} Jul 24 22:46:56.921902 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.921858 2571 scope.go:117] "RemoveContainer" containerID="16d233e3ab61becf7005c97ba2c0b15d0fef1b5d8722693daaeaa2fe5fb03d88" Jul 24 22:46:56.922258 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.922173 2571 scope.go:117] "RemoveContainer" containerID="89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11" Jul 24 22:46:56.922374 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:56.922340 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ffg2l_openshift-console-operator(3de1980f-7b67-4fd1-9981-4f952b4ee8b0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" podUID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" Jul 24 22:46:56.923335 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:56.923144 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9kzs\" (UniqueName: \"kubernetes.io/projected/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-api-access-f9kzs\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:57.418416 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:57.418365 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:57.418601 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:57.418480 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:46:57.418601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:57.418516 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:57.418601 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:57.418534 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:58.418520214 +0000 UTC m=+47.347859403 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : configmap "kube-rbac-proxy" not found Jul 24 22:46:57.418759 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:57.418604 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:46:57.418759 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:57.418665 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:46:58.418649773 +0000 UTC m=+47.347988966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : secret "insights-runtime-extractor-tls" not found Jul 24 22:46:57.926909 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:57.926882 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/1.log" Jul 24 22:46:57.927265 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:57.927182 2571 scope.go:117] "RemoveContainer" containerID="89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11" Jul 24 22:46:57.927346 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:57.927328 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-ffg2l_openshift-console-operator(3de1980f-7b67-4fd1-9981-4f952b4ee8b0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" podUID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" Jul 24 22:46:58.427869 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:58.427833 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:58.428069 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:58.427952 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:46:58.428069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:58.427966 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:46:58.428157 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:58.428080 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:00.428061088 +0000 UTC m=+49.357400277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : configmap "kube-rbac-proxy" not found Jul 24 22:46:58.428157 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:58.428105 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:46:58.428226 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:46:58.428159 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:00.428147171 +0000 UTC m=+49.357486360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : secret "insights-runtime-extractor-tls" not found Jul 24 22:46:58.557683 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:58.557657 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lzdj7_dae193ef-02ba-4de6-bc10-c521f0ee99e9/dns-node-resolver/0.log" Jul 24 22:46:59.557448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:46:59.557414 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tzrc7_bf4d6d27-e179-4ea2-b8fb-0172c057f2dd/node-ca/0.log" Jul 24 22:47:00.157308 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:00.157273 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk84c_f5d43aaf-5cc3-4545-8815-b91e60c4951f/serve-healthcheck-canary/0.log" Jul 24 22:47:00.446259 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:00.446187 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:00.446410 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:00.446310 2571 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 22:47:00.446410 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:00.446375 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:04.446360113 +0000 UTC m=+53.375699302 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : configmap "kube-rbac-proxy" not found Jul 24 22:47:00.446496 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:00.446476 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:00.446650 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:00.446626 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:47:00.446709 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:00.446698 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:04.446685463 +0000 UTC m=+53.376024693 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : secret "insights-runtime-extractor-tls" not found Jul 24 22:47:02.361195 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361159 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361209 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361304 2571 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361317 2571 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361355 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.361341867 +0000 UTC m=+67.290681059 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : secret "router-metrics-certs-default" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361411 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls podName:e18d6dea-7bf3-4e83-9f14-99ad865d33c8 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.361376794 +0000 UTC m=+67.290715987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls") pod "cluster-samples-operator-6748489656-8n9l8" (UID: "e18d6dea-7bf3-4e83-9f14-99ad865d33c8") : secret "samples-operator-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361478 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361505 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361523 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361543 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:02.361569 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361600 2571 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361634 2571 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361641 2571 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361649 2571 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64554999d9-mbc4d: secret "image-registry-tls" not found Jul 24 22:47:02.361670 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361669 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls podName:5092f984-2bce-4b55-a5c9-f3e69b726b07 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.36165447 +0000 UTC m=+67.290993672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-slws5" (UID: "5092f984-2bce-4b55-a5c9-f3e69b726b07") : secret "cluster-monitoring-operator-tls" not found Jul 24 22:47:02.362140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361686 2571 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 22:47:02.362140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361692 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls podName:e7d5de92-3fae-46c6-91dc-44c9d0404700 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.361679133 +0000 UTC m=+67.291018322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls") pod "image-registry-64554999d9-mbc4d" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700") : secret "image-registry-tls" not found Jul 24 22:47:02.362140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361708 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls podName:2e9c68a4-54dd-4b07-83b4-79cb055eb539 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.361700347 +0000 UTC m=+67.291039538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls") pod "dns-default-7cl5g" (UID: "2e9c68a4-54dd-4b07-83b4-79cb055eb539") : secret "dns-default-metrics-tls" not found Jul 24 22:47:02.362140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361725 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle podName:5eaa6fba-900c-4fb8-b51c-821f7112fd03 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.36171564 +0000 UTC m=+67.291054830 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle") pod "router-default-88d7b5c95-jqkth" (UID: "5eaa6fba-900c-4fb8-b51c-821f7112fd03") : configmap references non-existent config key: service-ca.crt Jul 24 22:47:02.362140 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:02.361738 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert podName:2d08fc13-992d-4c02-b9a1-a06ac9bec210 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:18.361731036 +0000 UTC m=+67.291070229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-bx5rd" (UID: "2d08fc13-992d-4c02-b9a1-a06ac9bec210") : secret "networking-console-plugin-cert" not found Jul 24 22:47:04.480930 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:04.480892 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:04.481307 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:04.480976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:04.481307 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:04.481033 2571 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 22:47:04.481307 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:04.481099 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls podName:d75bbedf-d9ba-4fd1-94c2-0349016e8c2a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:12.48108504 +0000 UTC m=+61.410424230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-c68zt" (UID: "d75bbedf-d9ba-4fd1-94c2-0349016e8c2a") : secret "insights-runtime-extractor-tls" not found Jul 24 22:47:04.481517 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:04.481457 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:06.840195 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.840155 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:06.840726 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.840216 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:06.840726 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.840594 2571 scope.go:117] "RemoveContainer" containerID="89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11" Jul 24 22:47:06.957994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.957970 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:47:06.958358 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.958345 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/1.log" Jul 24 22:47:06.958447 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.958377 2571 generic.go:356] "Generic (PLEG): container finished" podID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" containerID="f566b76a4342add9262bd2d0799ed48dd667dbf645ff240a56ba0f9e87d502d8" exitCode=255 Jul 24 22:47:06.958447 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.958425 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" event={"ID":"3de1980f-7b67-4fd1-9981-4f952b4ee8b0","Type":"ContainerDied","Data":"f566b76a4342add9262bd2d0799ed48dd667dbf645ff240a56ba0f9e87d502d8"} Jul 24 22:47:06.958547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.958498 2571 scope.go:117] "RemoveContainer" containerID="89183769b55945c6ddf7d56f3e7d3b80944a37f1bb0968b1e62c661882fe1e11" Jul 24 22:47:06.958802 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:06.958777 2571 scope.go:117] "RemoveContainer" containerID="f566b76a4342add9262bd2d0799ed48dd667dbf645ff240a56ba0f9e87d502d8" Jul 24 22:47:06.958995 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:06.958972 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-ffg2l_openshift-console-operator(3de1980f-7b67-4fd1-9981-4f952b4ee8b0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" podUID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" Jul 24 22:47:07.962144 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:07.962112 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:47:10.812589 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:10.812563 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-85sr5" Jul 24 22:47:12.554085 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.554021 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:12.556288 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.556260 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d75bbedf-d9ba-4fd1-94c2-0349016e8c2a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c68zt\" (UID: \"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a\") " pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:12.691058 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.691028 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlc68\"" Jul 24 22:47:12.700273 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.700251 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c68zt" Jul 24 22:47:12.845236 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.845161 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c68zt"] Jul 24 22:47:12.848032 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:12.848002 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd75bbedf_d9ba_4fd1_94c2_0349016e8c2a.slice/crio-7ede4183ef2af446d3135423745cf79c16e90b2bf9b7df82095968d34e023841 WatchSource:0}: Error finding container 7ede4183ef2af446d3135423745cf79c16e90b2bf9b7df82095968d34e023841: Status 404 returned error can't find the container with id 7ede4183ef2af446d3135423745cf79c16e90b2bf9b7df82095968d34e023841 Jul 24 22:47:12.977154 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:12.977119 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c68zt" event={"ID":"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a","Type":"ContainerStarted","Data":"7ede4183ef2af446d3135423745cf79c16e90b2bf9b7df82095968d34e023841"} Jul 24 22:47:16.839993 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:16.839962 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:16.840413 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:16.840024 2571 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:16.840413 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:16.840322 2571 scope.go:117] "RemoveContainer" containerID="f566b76a4342add9262bd2d0799ed48dd667dbf645ff240a56ba0f9e87d502d8" Jul 24 22:47:16.840522 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:16.840502 2571 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-ffg2l_openshift-console-operator(3de1980f-7b67-4fd1-9981-4f952b4ee8b0)\"" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" podUID="3de1980f-7b67-4fd1-9981-4f952b4ee8b0" Jul 24 22:47:17.298364 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.298308 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:47:17.298364 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.298368 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:47:17.298628 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.298439 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:47:17.300735 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.300713 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 22:47:17.301105 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.301089 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 22:47:17.301305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.301250 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtklb\" (UniqueName: \"kubernetes.io/projected/d89a4944-62c1-4d98-a707-12d2116d3d5d-kube-api-access-vtklb\") pod \"network-check-target-knh98\" (UID: \"d89a4944-62c1-4d98-a707-12d2116d3d5d\") " pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:47:17.311799 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.311775 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c870da1-f7fd-4241-a588-26b354f604fe-metrics-certs\") pod \"network-metrics-daemon-867cq\" (UID: \"3c870da1-f7fd-4241-a588-26b354f604fe\") " pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:47:17.311917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.311777 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c-original-pull-secret\") pod \"global-pull-secret-syncer-tvlk8\" (UID: \"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c\") " pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:47:17.565832 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.565765 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b4wx4\"" Jul 24 22:47:17.569302 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.569281 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-tvlk8" Jul 24 22:47:17.574635 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.574620 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:47:17.575216 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.575201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dmr4r\"" Jul 24 22:47:17.583997 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.583971 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-867cq" Jul 24 22:47:17.730461 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.730429 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-tvlk8"] Jul 24 22:47:17.734883 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:17.734856 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aa6fd6f_f248_4fb2_bffd_3c3c83345e0c.slice/crio-b30e833a34f217f61235b041d5e6ffed40bd5ca6c2177add1afdc52595748dee WatchSource:0}: Error finding container b30e833a34f217f61235b041d5e6ffed40bd5ca6c2177add1afdc52595748dee: Status 404 returned error can't find the container with id b30e833a34f217f61235b041d5e6ffed40bd5ca6c2177add1afdc52595748dee Jul 24 22:47:17.745891 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.745872 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knh98"] Jul 24 22:47:17.748171 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:17.748143 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd89a4944_62c1_4d98_a707_12d2116d3d5d.slice/crio-d1b52c9a510960a011934765f366982ff07a44b8ab863237cf2968541a2da1ac WatchSource:0}: Error finding container d1b52c9a510960a011934765f366982ff07a44b8ab863237cf2968541a2da1ac: Status 404 returned error can't find the container with id d1b52c9a510960a011934765f366982ff07a44b8ab863237cf2968541a2da1ac Jul 24 22:47:17.757776 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.757756 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-867cq"] Jul 24 22:47:17.760552 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:17.760523 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c870da1_f7fd_4241_a588_26b354f604fe.slice/crio-1cf8a72e8a8b0467924af4dcb79370a408f6f764d20d11901a65917b736e493b WatchSource:0}: Error finding container 1cf8a72e8a8b0467924af4dcb79370a408f6f764d20d11901a65917b736e493b: Status 404 returned error can't find the container with id 1cf8a72e8a8b0467924af4dcb79370a408f6f764d20d11901a65917b736e493b Jul 24 22:47:17.994484 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.994447 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c68zt" event={"ID":"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a","Type":"ContainerStarted","Data":"cf01b32ff54a026330e04f6138d380d08c6cbeab4012db501b47f8622ef2fbf5"} Jul 24 22:47:17.995701 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.995679 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-867cq" event={"ID":"3c870da1-f7fd-4241-a588-26b354f604fe","Type":"ContainerStarted","Data":"1cf8a72e8a8b0467924af4dcb79370a408f6f764d20d11901a65917b736e493b"} Jul 24 22:47:17.996810 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.996786 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvlk8" event={"ID":"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c","Type":"ContainerStarted","Data":"b30e833a34f217f61235b041d5e6ffed40bd5ca6c2177add1afdc52595748dee"} Jul 24 22:47:17.998056 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.998034 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knh98" event={"ID":"d89a4944-62c1-4d98-a707-12d2116d3d5d","Type":"ContainerStarted","Data":"ee03e94ae414817b2ed829f1414230fdae1ca6553e888700082bde10acfd4d32"} Jul 24 22:47:17.998134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.998071 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knh98" event={"ID":"d89a4944-62c1-4d98-a707-12d2116d3d5d","Type":"ContainerStarted","Data":"d1b52c9a510960a011934765f366982ff07a44b8ab863237cf2968541a2da1ac"} Jul 24 22:47:17.998134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:17.998115 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:47:18.012617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.012578 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-knh98" podStartSLOduration=66.012563123 podStartE2EDuration="1m6.012563123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:18.011974455 +0000 UTC m=+66.941313670" watchObservedRunningTime="2026-07-24 22:47:18.012563123 +0000 UTC m=+66.941902334" Jul 24 22:47:18.410119 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410068 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:18.410305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410151 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:47:18.410305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410186 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:18.410305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410213 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:18.410305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410239 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:18.410305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:47:18.411005 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.410793 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:47:18.411113 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.411088 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5eaa6fba-900c-4fb8-b51c-821f7112fd03-service-ca-bundle\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:18.413600 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.413569 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/e18d6dea-7bf3-4e83-9f14-99ad865d33c8-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-8n9l8\" (UID: \"e18d6dea-7bf3-4e83-9f14-99ad865d33c8\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:47:18.413704 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.413665 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2d08fc13-992d-4c02-b9a1-a06ac9bec210-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-bx5rd\" (UID: \"2d08fc13-992d-4c02-b9a1-a06ac9bec210\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:47:18.413770 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.413744 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5eaa6fba-900c-4fb8-b51c-821f7112fd03-metrics-certs\") pod \"router-default-88d7b5c95-jqkth\" (UID: \"5eaa6fba-900c-4fb8-b51c-821f7112fd03\") " pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:18.413770 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.413751 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/5092f984-2bce-4b55-a5c9-f3e69b726b07-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-slws5\" (UID: \"5092f984-2bce-4b55-a5c9-f3e69b726b07\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:47:18.414171 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.414149 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e9c68a4-54dd-4b07-83b4-79cb055eb539-metrics-tls\") pod \"dns-default-7cl5g\" (UID: \"2e9c68a4-54dd-4b07-83b4-79cb055eb539\") " pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:18.415532 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.415482 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"image-registry-64554999d9-mbc4d\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:18.580044 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.580013 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-59m4s\"" Jul 24 22:47:18.587972 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.587941 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:18.589428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.589379 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-ftpfs\"" Jul 24 22:47:18.598792 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.598764 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" Jul 24 22:47:18.610026 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.609832 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9xrkj\"" Jul 24 22:47:18.619032 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.618975 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" Jul 24 22:47:18.633477 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.633264 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-k5m62\"" Jul 24 22:47:18.641853 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.641776 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:18.649347 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.649154 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5flch\"" Jul 24 22:47:18.658012 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.657637 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" Jul 24 22:47:18.677263 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.677114 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-l2hpm\"" Jul 24 22:47:18.687418 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.686887 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:18.842888 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.842679 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:47:18.900489 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.896034 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8"] Jul 24 22:47:18.901114 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.901056 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5"] Jul 24 22:47:18.907434 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:18.907380 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5092f984_2bce_4b55_a5c9_f3e69b726b07.slice/crio-25dad90b6285ee34d665f651410de04c6111187778a17c3d2b108e51b0cb0782 WatchSource:0}: Error finding container 25dad90b6285ee34d665f651410de04c6111187778a17c3d2b108e51b0cb0782: Status 404 returned error can't find the container with id 25dad90b6285ee34d665f651410de04c6111187778a17c3d2b108e51b0cb0782 Jul 24 22:47:18.932835 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.932612 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-88d7b5c95-jqkth"] Jul 24 22:47:18.973365 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.973284 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7cl5g"] Jul 24 22:47:18.977029 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:18.976998 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e9c68a4_54dd_4b07_83b4_79cb055eb539.slice/crio-eda377498f6c617d790980b77c552cc7ae4f6a4aadd4b06a911be42bb2ad0731 WatchSource:0}: Error finding container eda377498f6c617d790980b77c552cc7ae4f6a4aadd4b06a911be42bb2ad0731: Status 404 returned error can't find the container with id eda377498f6c617d790980b77c552cc7ae4f6a4aadd4b06a911be42bb2ad0731 Jul 24 22:47:18.984156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:18.984132 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd"] Jul 24 22:47:18.988195 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:18.988142 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d08fc13_992d_4c02_b9a1_a06ac9bec210.slice/crio-52325e4a430353121aae307bce3bccbc95a25cef67fe18a19e5aa1e98c48a855 WatchSource:0}: Error finding container 52325e4a430353121aae307bce3bccbc95a25cef67fe18a19e5aa1e98c48a855: Status 404 returned error can't find the container with id 52325e4a430353121aae307bce3bccbc95a25cef67fe18a19e5aa1e98c48a855 Jul 24 22:47:19.009142 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.009065 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-88d7b5c95-jqkth" event={"ID":"5eaa6fba-900c-4fb8-b51c-821f7112fd03","Type":"ContainerStarted","Data":"7e934b3454d71310bda0cf146ed9fa1f533d7d0b14e586d009f778e1a7157640"} Jul 24 22:47:19.011904 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.011837 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" event={"ID":"e18d6dea-7bf3-4e83-9f14-99ad865d33c8","Type":"ContainerStarted","Data":"968bc5e8c0cf7b4af468c302d9f046449a3aee6232deb3063be941ae1b00a432"} Jul 24 22:47:19.013614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.013580 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" event={"ID":"5092f984-2bce-4b55-a5c9-f3e69b726b07","Type":"ContainerStarted","Data":"25dad90b6285ee34d665f651410de04c6111187778a17c3d2b108e51b0cb0782"} Jul 24 22:47:19.018495 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.018240 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" event={"ID":"e7d5de92-3fae-46c6-91dc-44c9d0404700","Type":"ContainerStarted","Data":"7d0ce15436b315278be2d2a5569d801bc4365c2d1c862d5558e92a3ccf0dd482"} Jul 24 22:47:19.021243 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.021175 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cl5g" event={"ID":"2e9c68a4-54dd-4b07-83b4-79cb055eb539","Type":"ContainerStarted","Data":"eda377498f6c617d790980b77c552cc7ae4f6a4aadd4b06a911be42bb2ad0731"} Jul 24 22:47:19.023125 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:19.022647 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" event={"ID":"2d08fc13-992d-4c02-b9a1-a06ac9bec210","Type":"ContainerStarted","Data":"52325e4a430353121aae307bce3bccbc95a25cef67fe18a19e5aa1e98c48a855"} Jul 24 22:47:20.035611 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.035575 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-88d7b5c95-jqkth" event={"ID":"5eaa6fba-900c-4fb8-b51c-821f7112fd03","Type":"ContainerStarted","Data":"19ae6d4f9ac2fef3d18ac3e05d6d81a045371a15d304b372b1f21b6fe364e828"} Jul 24 22:47:20.042567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.042463 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-867cq" event={"ID":"3c870da1-f7fd-4241-a588-26b354f604fe","Type":"ContainerStarted","Data":"153c3114c59b0239b211c04acc8b3198c916352928aa85424b7717aefc0e2e6a"} Jul 24 22:47:20.042567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.042529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-867cq" event={"ID":"3c870da1-f7fd-4241-a588-26b354f604fe","Type":"ContainerStarted","Data":"a74681d83330cb181b86f3efa8bb55a8d058b1950512c893562e7a4e1984dd4b"} Jul 24 22:47:20.046473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.045953 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" event={"ID":"e7d5de92-3fae-46c6-91dc-44c9d0404700","Type":"ContainerStarted","Data":"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a"} Jul 24 22:47:20.046473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.046421 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:20.061862 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.060662 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-88d7b5c95-jqkth" podStartSLOduration=65.060635154 podStartE2EDuration="1m5.060635154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:20.053766503 +0000 UTC m=+68.983105715" watchObservedRunningTime="2026-07-24 22:47:20.060635154 +0000 UTC m=+68.989974366" Jul 24 22:47:20.076892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.076562 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-867cq" podStartSLOduration=66.142834615 podStartE2EDuration="1m8.076543537s" totalImagesPullingTime="1.933708922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:20.075316163 +0000 UTC m=+69.004655375" watchObservedRunningTime="2026-07-24 22:47:20.076543537 +0000 UTC m=+69.005882752" Jul 24 22:47:20.092340 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.092230 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" podStartSLOduration=68.092210736 podStartE2EDuration="1m8.092210736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:20.091143857 +0000 UTC m=+69.020483068" watchObservedRunningTime="2026-07-24 22:47:20.092210736 +0000 UTC m=+69.021549948" Jul 24 22:47:20.642586 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.642441 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:20.646618 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:20.646548 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:21.051046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:21.051014 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:21.052372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:21.052349 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-88d7b5c95-jqkth" Jul 24 22:47:28.082824 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.082781 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cl5g" event={"ID":"2e9c68a4-54dd-4b07-83b4-79cb055eb539","Type":"ContainerStarted","Data":"43f3d9caed163b42971f263410ef2e4005fefe61e6606014caa3b7990ce7a6c1"} Jul 24 22:47:28.083341 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.082854 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7cl5g" event={"ID":"2e9c68a4-54dd-4b07-83b4-79cb055eb539","Type":"ContainerStarted","Data":"421b6dcc56f4b797c4417b48a522b7cb07afa1135f75f436a80d209fbce63331"} Jul 24 22:47:28.083341 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.082972 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:28.084659 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.084629 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" event={"ID":"2d08fc13-992d-4c02-b9a1-a06ac9bec210","Type":"ContainerStarted","Data":"9ababe5999013184c1e6e8812e9577c34658e67ef39241f7d5a64f48a4fc1a40"} Jul 24 22:47:28.086782 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.086758 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c68zt" event={"ID":"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a","Type":"ContainerStarted","Data":"830822bb161e11670293ff8d3020cfc95cbfc5540f337006d48abb259b8d95ff"} Jul 24 22:47:28.088700 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.088670 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" event={"ID":"e18d6dea-7bf3-4e83-9f14-99ad865d33c8","Type":"ContainerStarted","Data":"9003cd70bc4dc1a3e4a32d802c2f13fb3ee4790dffd3644dcafa418e2d21c0c2"} Jul 24 22:47:28.088826 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.088718 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" event={"ID":"e18d6dea-7bf3-4e83-9f14-99ad865d33c8","Type":"ContainerStarted","Data":"7d5dd54d3fd3f07286aab262e934c3afffa98ce4022199c604d9bbcae3f80136"} Jul 24 22:47:28.090281 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.090253 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-tvlk8" event={"ID":"1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c","Type":"ContainerStarted","Data":"5d59d21e737cecbd10e7389168dcfd9dae1da7a9732b4b46da09a54e7daf9472"} Jul 24 22:47:28.092046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.092025 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" event={"ID":"5092f984-2bce-4b55-a5c9-f3e69b726b07","Type":"ContainerStarted","Data":"78f11f3cbae20c338673d5cbb07d074df5bec86c68138fbfa22d8d114ed903d8"} Jul 24 22:47:28.119295 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.119237 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7cl5g" podStartSLOduration=34.087854173 podStartE2EDuration="42.119217105s" totalImagesPullingTime="8.031362932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:28.100839023 +0000 UTC m=+77.030178238" watchObservedRunningTime="2026-07-24 22:47:28.119217105 +0000 UTC m=+77.048556319" Jul 24 22:47:28.120212 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.120176 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-8n9l8" podStartSLOduration=65.106250809 podStartE2EDuration="1m13.120166107s" totalImagesPullingTime="8.013915298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:28.119203101 +0000 UTC m=+77.048542313" watchObservedRunningTime="2026-07-24 22:47:28.120166107 +0000 UTC m=+77.049505621" Jul 24 22:47:28.138639 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.138599 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-slws5" podStartSLOduration=65.038780809 podStartE2EDuration="1m13.13858463s" totalImagesPullingTime="8.099803821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:28.138134434 +0000 UTC m=+77.067473646" watchObservedRunningTime="2026-07-24 22:47:28.13858463 +0000 UTC m=+77.067923839" Jul 24 22:47:28.155977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.155911 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-bx5rd" podStartSLOduration=40.137084415 podStartE2EDuration="48.155889771s" totalImagesPullingTime="8.018805356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:28.154229971 +0000 UTC m=+77.083569181" watchObservedRunningTime="2026-07-24 22:47:28.155889771 +0000 UTC m=+77.085228985" Jul 24 22:47:28.173458 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:28.173384 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-tvlk8" podStartSLOduration=66.897551797 podStartE2EDuration="1m16.173365743s" totalImagesPullingTime="9.275813946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:28.17204485 +0000 UTC m=+77.101384062" watchObservedRunningTime="2026-07-24 22:47:28.173365743 +0000 UTC m=+77.102704957" Jul 24 22:47:31.105728 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:31.105690 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c68zt" event={"ID":"d75bbedf-d9ba-4fd1-94c2-0349016e8c2a","Type":"ContainerStarted","Data":"c466234fc2fef9a49b681fea8566f8d50a243790d318308cbda399067d2fca77"} Jul 24 22:47:31.122703 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:31.122643 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c68zt" podStartSLOduration=17.680409179 podStartE2EDuration="35.122626321s" totalImagesPullingTime="17.442217142s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 22:46:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:31.121568078 +0000 UTC m=+80.050907288" watchObservedRunningTime="2026-07-24 22:47:31.122626321 +0000 UTC m=+80.051965532" Jul 24 22:47:31.556841 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:31.556808 2571 scope.go:117] "RemoveContainer" containerID="f566b76a4342add9262bd2d0799ed48dd667dbf645ff240a56ba0f9e87d502d8" Jul 24 22:47:32.111964 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:32.111935 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:47:32.112381 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:32.112057 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" event={"ID":"3de1980f-7b67-4fd1-9981-4f952b4ee8b0","Type":"ContainerStarted","Data":"7a77199fad601028fa6d6c437d838c27b929a8d8d815c4dca254b53dcadd2b65"} Jul 24 22:47:32.112481 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:32.112410 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:32.131908 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:32.131857 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" podStartSLOduration=69.440893503 podStartE2EDuration="1m17.131844259s" totalImagesPullingTime="7.690950756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:46:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:32.131250644 +0000 UTC m=+81.060589867" watchObservedRunningTime="2026-07-24 22:47:32.131844259 +0000 UTC m=+81.061183470" Jul 24 22:47:32.151280 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:32.151254 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-ffg2l" Jul 24 22:47:38.009377 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.009249 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd"] Jul 24 22:47:38.013836 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.013817 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.016351 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.016331 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 22:47:38.016486 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.016330 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 22:47:38.016573 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.016557 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jgd2w\"" Jul 24 22:47:38.016795 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.016781 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 22:47:38.022268 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.022244 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd"] Jul 24 22:47:38.051482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.051447 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-wzlwv"] Jul 24 22:47:38.054712 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.054690 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.056767 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.056746 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 22:47:38.056887 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.056756 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jjfp8\"" Jul 24 22:47:38.057016 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.057002 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 22:47:38.057215 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.057201 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 22:47:38.080844 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080816 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-root\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080847 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080874 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847hw\" (UniqueName: \"kubernetes.io/projected/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-kube-api-access-847hw\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080903 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080943 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.080983 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.080983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-textfile\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081024 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-metrics-client-ca\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081057 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-sys\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081075 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6jv\" (UniqueName: \"kubernetes.io/projected/46bc6819-d493-49bd-b65b-449051088bd2-kube-api-access-4t6jv\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081099 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-wtmp\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081117 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.081209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.081183 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.098372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.098349 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7cl5g" Jul 24 22:47:38.182235 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182204 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.182235 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182241 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-root\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182260 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182306 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847hw\" (UniqueName: \"kubernetes.io/projected/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-kube-api-access-847hw\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182328 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-root\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182389 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182465 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-textfile\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-metrics-client-ca\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182519 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-sys\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182546 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6jv\" (UniqueName: \"kubernetes.io/projected/46bc6819-d493-49bd-b65b-449051088bd2-kube-api-access-4t6jv\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182582 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-wtmp\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182622 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.182628 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-sys\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.182921 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:38.182631 2571 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 22:47:38.183205 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:38.183003 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls podName:46bc6819-d493-49bd-b65b-449051088bd2 nodeName:}" failed. No retries permitted until 2026-07-24 22:47:38.682981584 +0000 UTC m=+87.612320788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls") pod "node-exporter-wzlwv" (UID: "46bc6819-d493-49bd-b65b-449051088bd2") : secret "node-exporter-tls" not found Jul 24 22:47:38.183205 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.183140 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-accelerators-collector-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.183339 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.183198 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-textfile\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.183339 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.183204 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-wtmp\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.183339 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.183293 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46bc6819-d493-49bd-b65b-449051088bd2-metrics-client-ca\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.183862 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.183837 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.184927 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.184903 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.185089 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.185072 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.185218 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.185199 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.192563 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.192534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6jv\" (UniqueName: \"kubernetes.io/projected/46bc6819-d493-49bd-b65b-449051088bd2-kube-api-access-4t6jv\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.193251 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.193227 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-847hw\" (UniqueName: \"kubernetes.io/projected/e3d8b8e1-2499-4bd1-ad2a-68146dbb599f-kube-api-access-847hw\") pod \"openshift-state-metrics-b8f49f5c6-scmkd\" (UID: \"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.324238 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.324207 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" Jul 24 22:47:38.469224 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.469195 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd"] Jul 24 22:47:38.472059 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:38.472032 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d8b8e1_2499_4bd1_ad2a_68146dbb599f.slice/crio-0e4658cc6c669d467438a48d0b4a64662cff2f0043fb62960702c7cc3a8174d8 WatchSource:0}: Error finding container 0e4658cc6c669d467438a48d0b4a64662cff2f0043fb62960702c7cc3a8174d8: Status 404 returned error can't find the container with id 0e4658cc6c669d467438a48d0b4a64662cff2f0043fb62960702c7cc3a8174d8 Jul 24 22:47:38.688506 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.688463 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.690714 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.690696 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46bc6819-d493-49bd-b65b-449051088bd2-node-exporter-tls\") pod \"node-exporter-wzlwv\" (UID: \"46bc6819-d493-49bd-b65b-449051088bd2\") " pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.964219 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:38.964140 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-wzlwv" Jul 24 22:47:38.976469 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:38.976436 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bc6819_d493_49bd_b65b_449051088bd2.slice/crio-fe574a6b4021b25119089442a50a1266e111e20727425d8547ebf76385bc4a0a WatchSource:0}: Error finding container fe574a6b4021b25119089442a50a1266e111e20727425d8547ebf76385bc4a0a: Status 404 returned error can't find the container with id fe574a6b4021b25119089442a50a1266e111e20727425d8547ebf76385bc4a0a Jul 24 22:47:39.093716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.093675 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:47:39.099562 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.099535 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.101631 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101605 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 22:47:39.101742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101642 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 22:47:39.101742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101681 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 22:47:39.101859 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101746 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 22:47:39.101907 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101862 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rgvjc\"" Jul 24 22:47:39.101965 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101900 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 22:47:39.101965 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101934 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 22:47:39.102058 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.101968 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 22:47:39.102058 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.102039 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 22:47:39.102155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.102134 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 22:47:39.110600 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.110578 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:47:39.138080 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.138042 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzlwv" event={"ID":"46bc6819-d493-49bd-b65b-449051088bd2","Type":"ContainerStarted","Data":"fe574a6b4021b25119089442a50a1266e111e20727425d8547ebf76385bc4a0a"} Jul 24 22:47:39.140307 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.140273 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" event={"ID":"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f","Type":"ContainerStarted","Data":"1651ae24e534b3c529abe8b938947b897ed6505b5afb0504ceb13f814cf7b296"} Jul 24 22:47:39.140462 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.140340 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" event={"ID":"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f","Type":"ContainerStarted","Data":"8b8462120c7f41a56739ebd68e9e65d44d998f95194a6872360f657e71299221"} Jul 24 22:47:39.140462 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.140361 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" event={"ID":"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f","Type":"ContainerStarted","Data":"0e4658cc6c669d467438a48d0b4a64662cff2f0043fb62960702c7cc3a8174d8"} Jul 24 22:47:39.194547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194476 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194489 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194509 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194575 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194621 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dhnt\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194787 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194824 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194881 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.194910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.194911 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296196 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296246 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296269 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296290 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296341 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296367 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296388 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dhnt\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296474 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296501 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296542 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296567 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.296787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.296609 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.298325 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:47:39.297529 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle podName:5ecd22c9-6bd2-48bc-bf06-2197be14b50a nodeName:}" failed. No retries permitted until 2026-07-24 22:47:39.797504571 +0000 UTC m=+88.726843773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a") : configmap references non-existent config key: ca-bundle.crt Jul 24 22:47:39.298325 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.297954 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.298325 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.297975 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.299316 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.299291 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.299794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.299771 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.301448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.301423 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.301518 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.301450 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.301601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.301578 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.301887 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.301838 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.301887 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.301849 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.302209 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.302185 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.303238 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.303216 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.305656 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.305631 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dhnt\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.801147 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.801111 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:39.801994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:39.801974 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:40.010824 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.010792 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:47:40.148601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.148565 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" event={"ID":"e3d8b8e1-2499-4bd1-ad2a-68146dbb599f","Type":"ContainerStarted","Data":"a41eccffa3f40789bb1902afbf936c70156d45bfd3a4733cd40a6b5eeeb3d2b7"} Jul 24 22:47:40.150151 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.150127 2571 generic.go:356] "Generic (PLEG): container finished" podID="46bc6819-d493-49bd-b65b-449051088bd2" containerID="e6e40e06c5bde7ce3661e90dceffdd27c7f1a2960560887114b6a10d182b643a" exitCode=0 Jul 24 22:47:40.150312 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.150160 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzlwv" event={"ID":"46bc6819-d493-49bd-b65b-449051088bd2","Type":"ContainerDied","Data":"e6e40e06c5bde7ce3661e90dceffdd27c7f1a2960560887114b6a10d182b643a"} Jul 24 22:47:40.155288 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.155264 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:47:40.158480 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:40.158457 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecd22c9_6bd2_48bc_bf06_2197be14b50a.slice/crio-a8a33cba6f74fbee61f2cc2af52cd49750d2eb2aab0d46fdbdd4fddae2b1c00a WatchSource:0}: Error finding container a8a33cba6f74fbee61f2cc2af52cd49750d2eb2aab0d46fdbdd4fddae2b1c00a: Status 404 returned error can't find the container with id a8a33cba6f74fbee61f2cc2af52cd49750d2eb2aab0d46fdbdd4fddae2b1c00a Jul 24 22:47:40.166548 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:40.166504 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-scmkd" podStartSLOduration=1.859737441 podStartE2EDuration="3.166488307s" totalImagesPullingTime="1.306750866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:40.164905757 +0000 UTC m=+89.094244968" watchObservedRunningTime="2026-07-24 22:47:40.166488307 +0000 UTC m=+89.095827518" Jul 24 22:47:41.160415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.160368 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"a8a33cba6f74fbee61f2cc2af52cd49750d2eb2aab0d46fdbdd4fddae2b1c00a"} Jul 24 22:47:41.162576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.162551 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzlwv" event={"ID":"46bc6819-d493-49bd-b65b-449051088bd2","Type":"ContainerStarted","Data":"36ba16ff48987758a1a8fd2b4bd0c222396b159620b0b23a9dc1af55140bd68f"} Jul 24 22:47:41.162696 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.162598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-wzlwv" event={"ID":"46bc6819-d493-49bd-b65b-449051088bd2","Type":"ContainerStarted","Data":"1192fa237db27e7d87feebb48bfe1d3dcd75f02c835b7d08d35425f52b68277c"} Jul 24 22:47:41.181901 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.181839 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-wzlwv" podStartSLOduration=2.1959384379999998 podStartE2EDuration="3.181817062s" totalImagesPullingTime="917.233849ms" totalInitContainerRuntime="68.644775ms" isStatefulPod=true podCreationTimestamp="2026-07-24 22:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:41.180415087 +0000 UTC m=+90.109754308" watchObservedRunningTime="2026-07-24 22:47:41.181817062 +0000 UTC m=+90.111156274" Jul 24 22:47:41.254409 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.254359 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:47:41.257854 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.257832 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.259721 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.259688 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 22:47:41.259838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.259746 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 22:47:41.259838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.259695 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 22:47:41.259838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.259695 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 22:47:41.259838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.259825 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 22:47:41.260153 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.260131 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 22:47:41.260306 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.260135 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 22:47:41.260389 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.260372 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5s8t\"" Jul 24 22:47:41.270585 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.270558 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:47:41.316208 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316162 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.316208 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.316443 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316235 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb89g\" (UniqueName: \"kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.316443 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316266 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.316443 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316363 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.316443 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.316428 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417492 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417413 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417492 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417470 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417492 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417496 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417512 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb89g\" (UniqueName: \"kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417559 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.417716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.417600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.418230 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.418207 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.418330 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.418248 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.418330 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.418287 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.419994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.419967 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.420089 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.420020 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.425716 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.425694 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb89g\" (UniqueName: \"kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g\") pod \"console-6655d74844-85krt\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.568852 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.568813 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:41.697349 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:41.697271 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:47:41.700422 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:41.700377 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb69350f_3924_4b6a_a52d_bcf40d589078.slice/crio-e81a12f97d0ebfb5eb1c59dce1fad423ab0edecb590dd0df7cb927fa6089ab44 WatchSource:0}: Error finding container e81a12f97d0ebfb5eb1c59dce1fad423ab0edecb590dd0df7cb927fa6089ab44: Status 404 returned error can't find the container with id e81a12f97d0ebfb5eb1c59dce1fad423ab0edecb590dd0df7cb927fa6089ab44 Jul 24 22:47:42.060259 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.060230 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:47:42.168090 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.168051 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="db65b520a9dadb519c60f3bd5b0efcd2a05d54ca4498db9125ffd4c67fb10e08" exitCode=0 Jul 24 22:47:42.168516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.168136 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"db65b520a9dadb519c60f3bd5b0efcd2a05d54ca4498db9125ffd4c67fb10e08"} Jul 24 22:47:42.169457 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.169428 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6655d74844-85krt" event={"ID":"bb69350f-3924-4b6a-a52d-bcf40d589078","Type":"ContainerStarted","Data":"e81a12f97d0ebfb5eb1c59dce1fad423ab0edecb590dd0df7cb927fa6089ab44"} Jul 24 22:47:42.345668 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.345585 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f58db8566-zmzls"] Jul 24 22:47:42.349570 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.349543 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.352485 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352305 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 22:47:42.352485 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352378 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 22:47:42.352691 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352672 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 22:47:42.352767 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352749 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-5se8ne0rtidvu\"" Jul 24 22:47:42.352801 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352775 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-fb2h7\"" Jul 24 22:47:42.352840 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.352792 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 22:47:42.359731 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.359704 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f58db8566-zmzls"] Jul 24 22:47:42.428568 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-metrics-server-audit-profiles\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.428765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428595 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjtdl\" (UniqueName: \"kubernetes.io/projected/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-kube-api-access-pjtdl\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.428765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428656 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-client-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.428765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428679 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-audit-log\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.428765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428711 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-client-certs\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.428765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428747 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-tls\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.429029 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.428773 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529269 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-metrics-server-audit-profiles\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529293 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjtdl\" (UniqueName: \"kubernetes.io/projected/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-kube-api-access-pjtdl\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529340 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-client-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529360 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-audit-log\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529403 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-client-certs\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529435 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-tls\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.529583 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.529468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.530169 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.530143 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.530472 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.530430 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-audit-log\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.530592 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.530534 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-metrics-server-audit-profiles\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.533031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.533003 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-tls\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.533031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.533025 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-client-ca-bundle\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.533210 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.533100 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-secret-metrics-server-client-certs\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.536533 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.536492 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjtdl\" (UniqueName: \"kubernetes.io/projected/e16a8a9d-bea3-4ed7-a148-c32af05fdf0d-kube-api-access-pjtdl\") pod \"metrics-server-5f58db8566-zmzls\" (UID: \"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d\") " pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.662151 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.662070 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:47:42.856022 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:42.855983 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f58db8566-zmzls"] Jul 24 22:47:42.862899 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:42.862865 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode16a8a9d_bea3_4ed7_a148_c32af05fdf0d.slice/crio-19c3826ce1387378c3ea684d50e47826f01834a2a6679066282e49006cb8e2b2 WatchSource:0}: Error finding container 19c3826ce1387378c3ea684d50e47826f01834a2a6679066282e49006cb8e2b2: Status 404 returned error can't find the container with id 19c3826ce1387378c3ea684d50e47826f01834a2a6679066282e49006cb8e2b2 Jul 24 22:47:43.178535 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:43.178494 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" event={"ID":"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d","Type":"ContainerStarted","Data":"19c3826ce1387378c3ea684d50e47826f01834a2a6679066282e49006cb8e2b2"} Jul 24 22:47:44.186535 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.186496 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"6cedbc9e34d36a09e6eeb94f7a01f1e467deaf80cccf4cab75d2f7aa492ea183"} Jul 24 22:47:44.187011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.186584 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"3870f380a049450c3a1d8cee607239863367650d6a584a799cc82891a85fdc3c"} Jul 24 22:47:44.187011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.186628 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"5c7bfc969148bbe0392223aa9af3bdc139b858e3957fc2c427515cd2032f1c7e"} Jul 24 22:47:44.187011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.186671 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"3cabeffed3d3e3fa242bd20a8cd3cfa83ddfdf4fb443dcf1ba730666f14a4046"} Jul 24 22:47:44.187011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.186716 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"bac0bc8a4627120ecbcf07ee287834a1879c723ad746553684009b92acb4dc83"} Jul 24 22:47:44.258515 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.258375 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:47:44.262684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.262661 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.264793 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.264770 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2snj6\"" Jul 24 22:47:44.264917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.264777 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 22:47:44.264917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.264777 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 22:47:44.265155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.265137 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 22:47:44.265656 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.265636 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 22:47:44.265927 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.265913 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 22:47:44.266022 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.265972 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 22:47:44.266022 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266011 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7edg8tqfbn0f5\"" Jul 24 22:47:44.266269 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266245 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 22:47:44.266357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266303 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 22:47:44.266357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266336 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 22:47:44.266632 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266598 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 22:47:44.266751 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266672 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 22:47:44.266838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.266822 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 22:47:44.267021 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.267005 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 22:47:44.268520 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.268502 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 22:47:44.280158 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.280133 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:47:44.346711 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346745 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346771 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346802 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346828 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346854 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346893 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.346934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346933 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.346973 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347002 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svptp\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347043 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347070 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347098 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347127 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347153 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347181 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347212 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347243 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347270 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.347319 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.347304 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448106 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448143 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448165 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448350 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448410 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448456 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448503 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448533 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svptp\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448579 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448606 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448608 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448666 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448694 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448723 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448754 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448784 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448813 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.448900 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448889 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.449308 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.448933 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.449926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.450277 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.450582 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.451118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451771 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.451520 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.451832 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.451789 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.452442 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.451998 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.452442 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.452271 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.452442 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.452371 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.452663 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.452592 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.452897 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.452871 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.453780 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.453735 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.454310 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.454185 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.454310 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.454244 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.455834 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.455784 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.455948 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.455919 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.456010 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.455948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.456871 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.456850 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svptp\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp\") pod \"prometheus-k8s-0\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:44.574469 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:44.574432 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:45.423841 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:45.423807 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:47:45.892005 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:45.891959 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:47:45.897051 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:47:45.897010 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4631710d_7e8c_4dd2_ad18_4aad6c9faa15.slice/crio-b1a989df30ab735fd6712bed04e541b5a495b9706bcc007fd9c1ea4ca8f4ec07 WatchSource:0}: Error finding container b1a989df30ab735fd6712bed04e541b5a495b9706bcc007fd9c1ea4ca8f4ec07: Status 404 returned error can't find the container with id b1a989df30ab735fd6712bed04e541b5a495b9706bcc007fd9c1ea4ca8f4ec07 Jul 24 22:47:46.197242 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.197203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" event={"ID":"e16a8a9d-bea3-4ed7-a148-c32af05fdf0d","Type":"ContainerStarted","Data":"0aad29b383dd1843879e6fa4bdc340521d9da0069e7da4a7fba100a12a74bcdb"} Jul 24 22:47:46.198896 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.198862 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6655d74844-85krt" event={"ID":"bb69350f-3924-4b6a-a52d-bcf40d589078","Type":"ContainerStarted","Data":"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a"} Jul 24 22:47:46.200505 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.200479 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" exitCode=0 Jul 24 22:47:46.200613 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.200552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} Jul 24 22:47:46.200613 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.200601 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"b1a989df30ab735fd6712bed04e541b5a495b9706bcc007fd9c1ea4ca8f4ec07"} Jul 24 22:47:46.204311 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.204292 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerStarted","Data":"427ff85e445965b6263c769db5567fd1489d730011a85da580dce8f869f2c6ed"} Jul 24 22:47:46.212545 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.212505 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" podStartSLOduration=1.32663779 podStartE2EDuration="4.212489945s" totalImagesPullingTime="2.885852155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:46.211984948 +0000 UTC m=+95.141324162" watchObservedRunningTime="2026-07-24 22:47:46.212489945 +0000 UTC m=+95.141829157" Jul 24 22:47:46.237081 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.237037 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.697417149 podStartE2EDuration="7.237019633s" totalImagesPullingTime="4.520519624s" totalInitContainerRuntime="19.08286ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:46.23495428 +0000 UTC m=+95.164293492" watchObservedRunningTime="2026-07-24 22:47:46.237019633 +0000 UTC m=+95.166358845" Jul 24 22:47:46.250596 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:46.250547 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6655d74844-85krt" podStartSLOduration=1.198881595 podStartE2EDuration="5.250533075s" totalImagesPullingTime="4.05165148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:46.24985097 +0000 UTC m=+95.179190182" watchObservedRunningTime="2026-07-24 22:47:46.250533075 +0000 UTC m=+95.179872300" Jul 24 22:47:49.025714 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:49.025653 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-knh98" Jul 24 22:47:50.227643 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:50.227559 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} Jul 24 22:47:50.227643 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:50.227628 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} Jul 24 22:47:51.569445 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:51.569413 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:51.569867 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:51.569475 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:51.574432 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:51.574406 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:52.244813 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.244708 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} Jul 24 22:47:52.244994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.244822 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} Jul 24 22:47:52.244994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.244862 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} Jul 24 22:47:52.244994 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.244896 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerStarted","Data":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} Jul 24 22:47:52.248546 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.248520 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:47:52.283848 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:52.283782 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.137367571 podStartE2EDuration="8.283756757s" totalImagesPullingTime="5.122889116s" totalInitContainerRuntime="23.50007ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:47:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:47:52.280994197 +0000 UTC m=+101.210333430" watchObservedRunningTime="2026-07-24 22:47:52.283756757 +0000 UTC m=+101.213095970" Jul 24 22:47:54.574618 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:54.574583 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:47:59.344763 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:47:59.344728 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:48:02.663502 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:02.663462 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:48:02.663888 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:02.663529 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:48:10.445167 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.445116 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" podUID="e7d5de92-3fae-46c6-91dc-44c9d0404700" containerName="registry" containerID="cri-o://31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a" gracePeriod=30 Jul 24 22:48:10.691089 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.691061 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:48:10.802048 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802020 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5lrn\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802202 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802057 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802202 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802089 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802202 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802115 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802202 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802180 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802440 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802248 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802440 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802278 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.802440 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802312 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca\") pod \"e7d5de92-3fae-46c6-91dc-44c9d0404700\" (UID: \"e7d5de92-3fae-46c6-91dc-44c9d0404700\") " Jul 24 22:48:10.803002 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.802970 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:10.803110 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.803016 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:10.804806 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.804777 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn" (OuterVolumeSpecName: "kube-api-access-c5lrn") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "kube-api-access-c5lrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:10.804928 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.804895 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:10.804998 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.804914 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:10.804998 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.804970 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:10.805523 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.805494 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:10.811181 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.811156 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e7d5de92-3fae-46c6-91dc-44c9d0404700" (UID: "e7d5de92-3fae-46c6-91dc-44c9d0404700"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:48:10.903200 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903165 2571 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-image-registry-private-configuration\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903200 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903198 2571 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-bound-sa-token\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903210 2571 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7d5de92-3fae-46c6-91dc-44c9d0404700-installation-pull-secrets\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903220 2571 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-trusted-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903228 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5lrn\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-kube-api-access-c5lrn\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903236 2571 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903244 2571 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7d5de92-3fae-46c6-91dc-44c9d0404700-ca-trust-extracted\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:10.903388 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:10.903252 2571 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7d5de92-3fae-46c6-91dc-44c9d0404700-registry-certificates\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:11.323721 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.323687 2571 generic.go:356] "Generic (PLEG): container finished" podID="dfcc5b79-9279-4686-a36a-93fc1b62d587" containerID="ba296f131ebce1e6c12af85d56ccfc6adba5eb670d558439d84d3d7f9e4898e7" exitCode=0 Jul 24 22:48:11.323904 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.323763 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" event={"ID":"dfcc5b79-9279-4686-a36a-93fc1b62d587","Type":"ContainerDied","Data":"ba296f131ebce1e6c12af85d56ccfc6adba5eb670d558439d84d3d7f9e4898e7"} Jul 24 22:48:11.324156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.324133 2571 scope.go:117] "RemoveContainer" containerID="ba296f131ebce1e6c12af85d56ccfc6adba5eb670d558439d84d3d7f9e4898e7" Jul 24 22:48:11.325053 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.325027 2571 generic.go:356] "Generic (PLEG): container finished" podID="e7d5de92-3fae-46c6-91dc-44c9d0404700" containerID="31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a" exitCode=0 Jul 24 22:48:11.325155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.325083 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" event={"ID":"e7d5de92-3fae-46c6-91dc-44c9d0404700","Type":"ContainerDied","Data":"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a"} Jul 24 22:48:11.325155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.325096 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" Jul 24 22:48:11.325155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.325124 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64554999d9-mbc4d" event={"ID":"e7d5de92-3fae-46c6-91dc-44c9d0404700","Type":"ContainerDied","Data":"7d0ce15436b315278be2d2a5569d801bc4365c2d1c862d5558e92a3ccf0dd482"} Jul 24 22:48:11.325155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.325151 2571 scope.go:117] "RemoveContainer" containerID="31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a" Jul 24 22:48:11.338739 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.337098 2571 scope.go:117] "RemoveContainer" containerID="31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a" Jul 24 22:48:11.339038 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:48:11.339005 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a\": container with ID starting with 31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a not found: ID does not exist" containerID="31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a" Jul 24 22:48:11.339130 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.339051 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a"} err="failed to get container status \"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a\": rpc error: code = NotFound desc = could not find container \"31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a\": container with ID starting with 31ab01bf8b57096e49e0eda477408a32580e3f3caeca3cbc52d7d95990cbe60a not found: ID does not exist" Jul 24 22:48:11.350478 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.350375 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:48:11.356696 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.356669 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64554999d9-mbc4d"] Jul 24 22:48:11.559742 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:11.559682 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7d5de92-3fae-46c6-91dc-44c9d0404700" path="/var/lib/kubelet/pods/e7d5de92-3fae-46c6-91dc-44c9d0404700/volumes" Jul 24 22:48:12.331356 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:12.331316 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-mmxlv" event={"ID":"dfcc5b79-9279-4686-a36a-93fc1b62d587","Type":"ContainerStarted","Data":"650fcd2a0df350c85683c7ea6cfd6809f030bde8f54a9b4aad955ca00677fec6"} Jul 24 22:48:16.348539 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.348499 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ab34ede-a8e4-4faa-8447-c3ae1b89b77c" containerID="1dfe950c3979e9b3e190496084516940e222e3a73ea9d4a0d6ff871d597622d1" exitCode=0 Jul 24 22:48:16.348972 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.348576 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" event={"ID":"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c","Type":"ContainerDied","Data":"1dfe950c3979e9b3e190496084516940e222e3a73ea9d4a0d6ff871d597622d1"} Jul 24 22:48:16.349034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.348970 2571 scope.go:117] "RemoveContainer" containerID="1dfe950c3979e9b3e190496084516940e222e3a73ea9d4a0d6ff871d597622d1" Jul 24 22:48:16.350149 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.350129 2571 generic.go:356] "Generic (PLEG): container finished" podID="69cd7378-d549-481c-b80a-5f5f7e140c51" containerID="ad9fd25545afa01b01b8ad59ecb0644e4f433aabb63926fff6c61dfcca2961cf" exitCode=0 Jul 24 22:48:16.350226 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.350200 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" event={"ID":"69cd7378-d549-481c-b80a-5f5f7e140c51","Type":"ContainerDied","Data":"ad9fd25545afa01b01b8ad59ecb0644e4f433aabb63926fff6c61dfcca2961cf"} Jul 24 22:48:16.350519 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:16.350506 2571 scope.go:117] "RemoveContainer" containerID="ad9fd25545afa01b01b8ad59ecb0644e4f433aabb63926fff6c61dfcca2961cf" Jul 24 22:48:17.358760 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:17.358710 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2" event={"ID":"5ab34ede-a8e4-4faa-8447-c3ae1b89b77c","Type":"ContainerStarted","Data":"f87c6177ebaf4f63e02fd40acbe018ce98c25232db4af058a54e53a56ff7acc6"} Jul 24 22:48:17.360862 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:17.360833 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-mp6t4" event={"ID":"69cd7378-d549-481c-b80a-5f5f7e140c51","Type":"ContainerStarted","Data":"f0167e39609481f5c6d8850a6c3bd63cab74814b6498a88868c9866ec6ad35f1"} Jul 24 22:48:22.668458 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:22.668388 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:48:22.672761 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:22.672736 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f58db8566-zmzls" Jul 24 22:48:24.364897 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.364846 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6655d74844-85krt" podUID="bb69350f-3924-4b6a-a52d-bcf40d589078" containerName="console" containerID="cri-o://9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a" gracePeriod=15 Jul 24 22:48:24.653149 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.653114 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:48:24.712788 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712759 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.712977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712799 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.712977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712826 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bb89g\" (UniqueName: \"kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.712977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712889 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.712977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712951 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.713189 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.712982 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca\") pod \"bb69350f-3924-4b6a-a52d-bcf40d589078\" (UID: \"bb69350f-3924-4b6a-a52d-bcf40d589078\") " Jul 24 22:48:24.713244 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.713176 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:24.713244 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.713217 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config" (OuterVolumeSpecName: "console-config") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:24.713491 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.713457 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca" (OuterVolumeSpecName: "service-ca") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:24.715141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.715106 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:24.715141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.715125 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:24.715289 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.715223 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g" (OuterVolumeSpecName: "kube-api-access-bb89g") pod "bb69350f-3924-4b6a-a52d-bcf40d589078" (UID: "bb69350f-3924-4b6a-a52d-bcf40d589078"). InnerVolumeSpecName "kube-api-access-bb89g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:24.814234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814189 2571 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-oauth-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:24.814234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814227 2571 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:24.814234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814238 2571 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-oauth-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:24.814234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814246 2571 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bb69350f-3924-4b6a-a52d-bcf40d589078-console-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:24.814527 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814255 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb89g\" (UniqueName: \"kubernetes.io/projected/bb69350f-3924-4b6a-a52d-bcf40d589078-kube-api-access-bb89g\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:24.814527 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:24.814265 2571 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bb69350f-3924-4b6a-a52d-bcf40d589078-console-serving-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:25.396279 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.396240 2571 generic.go:356] "Generic (PLEG): container finished" podID="bb69350f-3924-4b6a-a52d-bcf40d589078" containerID="9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a" exitCode=0 Jul 24 22:48:25.396720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.396292 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6655d74844-85krt" Jul 24 22:48:25.396720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.396329 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6655d74844-85krt" event={"ID":"bb69350f-3924-4b6a-a52d-bcf40d589078","Type":"ContainerDied","Data":"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a"} Jul 24 22:48:25.396720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.396388 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6655d74844-85krt" event={"ID":"bb69350f-3924-4b6a-a52d-bcf40d589078","Type":"ContainerDied","Data":"e81a12f97d0ebfb5eb1c59dce1fad423ab0edecb590dd0df7cb927fa6089ab44"} Jul 24 22:48:25.396720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.396429 2571 scope.go:117] "RemoveContainer" containerID="9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a" Jul 24 22:48:25.405703 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.405686 2571 scope.go:117] "RemoveContainer" containerID="9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a" Jul 24 22:48:25.405955 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:48:25.405936 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a\": container with ID starting with 9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a not found: ID does not exist" containerID="9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a" Jul 24 22:48:25.406005 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.405963 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a"} err="failed to get container status \"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a\": rpc error: code = NotFound desc = could not find container \"9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a\": container with ID starting with 9476c59850d2a13ff2a8dee11c6f531529298aaedfd883b3e2d683ed4809c83a not found: ID does not exist" Jul 24 22:48:25.417556 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.417532 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:48:25.422614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.422592 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6655d74844-85krt"] Jul 24 22:48:25.559769 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:25.559725 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb69350f-3924-4b6a-a52d-bcf40d589078" path="/var/lib/kubelet/pods/bb69350f-3924-4b6a-a52d-bcf40d589078/volumes" Jul 24 22:48:44.574919 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:44.574878 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:48:44.590872 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:44.590847 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:48:45.497694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:45.497663 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:48:58.315956 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.315914 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:48:58.316490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316322 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="alertmanager" containerID="cri-o://bac0bc8a4627120ecbcf07ee287834a1879c723ad746553684009b92acb4dc83" gracePeriod=120 Jul 24 22:48:58.316490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316363 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-metric" containerID="cri-o://6cedbc9e34d36a09e6eeb94f7a01f1e467deaf80cccf4cab75d2f7aa492ea183" gracePeriod=120 Jul 24 22:48:58.316490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316424 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="config-reloader" containerID="cri-o://3cabeffed3d3e3fa242bd20a8cd3cfa83ddfdf4fb443dcf1ba730666f14a4046" gracePeriod=120 Jul 24 22:48:58.316674 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316488 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="prom-label-proxy" containerID="cri-o://427ff85e445965b6263c769db5567fd1489d730011a85da580dce8f869f2c6ed" gracePeriod=120 Jul 24 22:48:58.316674 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316528 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy" containerID="cri-o://3870f380a049450c3a1d8cee607239863367650d6a584a799cc82891a85fdc3c" gracePeriod=120 Jul 24 22:48:58.316776 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.316669 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-web" containerID="cri-o://5c7bfc969148bbe0392223aa9af3bdc139b858e3957fc2c427515cd2032f1c7e" gracePeriod=120 Jul 24 22:48:58.544959 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.544914 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="427ff85e445965b6263c769db5567fd1489d730011a85da580dce8f869f2c6ed" exitCode=0 Jul 24 22:48:58.544959 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.544954 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="3870f380a049450c3a1d8cee607239863367650d6a584a799cc82891a85fdc3c" exitCode=0 Jul 24 22:48:58.544959 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.544966 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="3cabeffed3d3e3fa242bd20a8cd3cfa83ddfdf4fb443dcf1ba730666f14a4046" exitCode=0 Jul 24 22:48:58.545180 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.544975 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="bac0bc8a4627120ecbcf07ee287834a1879c723ad746553684009b92acb4dc83" exitCode=0 Jul 24 22:48:58.545180 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.544983 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"427ff85e445965b6263c769db5567fd1489d730011a85da580dce8f869f2c6ed"} Jul 24 22:48:58.545180 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.545081 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"3870f380a049450c3a1d8cee607239863367650d6a584a799cc82891a85fdc3c"} Jul 24 22:48:58.545180 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.545109 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"3cabeffed3d3e3fa242bd20a8cd3cfa83ddfdf4fb443dcf1ba730666f14a4046"} Jul 24 22:48:58.545180 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:58.545132 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"bac0bc8a4627120ecbcf07ee287834a1879c723ad746553684009b92acb4dc83"} Jul 24 22:48:59.553295 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.553184 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="6cedbc9e34d36a09e6eeb94f7a01f1e467deaf80cccf4cab75d2f7aa492ea183" exitCode=0 Jul 24 22:48:59.553295 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.553207 2571 generic.go:356] "Generic (PLEG): container finished" podID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerID="5c7bfc969148bbe0392223aa9af3bdc139b858e3957fc2c427515cd2032f1c7e" exitCode=0 Jul 24 22:48:59.553295 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.553254 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"6cedbc9e34d36a09e6eeb94f7a01f1e467deaf80cccf4cab75d2f7aa492ea183"} Jul 24 22:48:59.553725 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.553322 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"5c7bfc969148bbe0392223aa9af3bdc139b858e3957fc2c427515cd2032f1c7e"} Jul 24 22:48:59.574765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.574223 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:48:59.634131 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634091 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634131 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634133 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634158 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634184 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634217 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634335 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634380 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dhnt\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634448 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634474 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634501 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634534 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634562 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:48:59.634622 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634588 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.635037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634634 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\" (UID: \"5ecd22c9-6bd2-48bc-bf06-2197be14b50a\") " Jul 24 22:48:59.635037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634775 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:59.635037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634973 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-main-db\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.635037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.634993 2571 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-metrics-client-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.635667 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.635331 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:48:59.637008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.636983 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.638339 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.638312 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt" (OuterVolumeSpecName: "kube-api-access-2dhnt") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "kube-api-access-2dhnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:59.638464 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.638438 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume" (OuterVolumeSpecName: "config-volume") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.638585 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.638543 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.638825 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.638798 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:48:59.638977 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.638948 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out" (OuterVolumeSpecName: "config-out") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:48:59.639255 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.639186 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.639743 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.639718 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.643065 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.642954 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.649973 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.649952 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config" (OuterVolumeSpecName: "web-config") pod "5ecd22c9-6bd2-48bc-bf06-2197be14b50a" (UID: "5ecd22c9-6bd2-48bc-bf06-2197be14b50a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:48:59.735473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735437 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-out\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735467 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-web-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735473 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735475 2571 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735487 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735498 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dhnt\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-kube-api-access-2dhnt\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735509 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735518 2571 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-cluster-tls-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735526 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-tls-assets\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735534 2571 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735542 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-main-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:48:59.735694 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:48:59.735551 2571 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5ecd22c9-6bd2-48bc-bf06-2197be14b50a-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:00.561142 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.561105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5ecd22c9-6bd2-48bc-bf06-2197be14b50a","Type":"ContainerDied","Data":"a8a33cba6f74fbee61f2cc2af52cd49750d2eb2aab0d46fdbdd4fddae2b1c00a"} Jul 24 22:49:00.561681 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.561178 2571 scope.go:117] "RemoveContainer" containerID="427ff85e445965b6263c769db5567fd1489d730011a85da580dce8f869f2c6ed" Jul 24 22:49:00.561681 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.561178 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.569043 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.569016 2571 scope.go:117] "RemoveContainer" containerID="6cedbc9e34d36a09e6eeb94f7a01f1e467deaf80cccf4cab75d2f7aa492ea183" Jul 24 22:49:00.576095 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.576077 2571 scope.go:117] "RemoveContainer" containerID="3870f380a049450c3a1d8cee607239863367650d6a584a799cc82891a85fdc3c" Jul 24 22:49:00.582935 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.582915 2571 scope.go:117] "RemoveContainer" containerID="5c7bfc969148bbe0392223aa9af3bdc139b858e3957fc2c427515cd2032f1c7e" Jul 24 22:49:00.583839 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.583818 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:49:00.591112 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.591096 2571 scope.go:117] "RemoveContainer" containerID="3cabeffed3d3e3fa242bd20a8cd3cfa83ddfdf4fb443dcf1ba730666f14a4046" Jul 24 22:49:00.591284 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.591264 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:49:00.598094 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.598079 2571 scope.go:117] "RemoveContainer" containerID="bac0bc8a4627120ecbcf07ee287834a1879c723ad746553684009b92acb4dc83" Jul 24 22:49:00.604847 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.604830 2571 scope.go:117] "RemoveContainer" containerID="db65b520a9dadb519c60f3bd5b0efcd2a05d54ca4498db9125ffd4c67fb10e08" Jul 24 22:49:00.618069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618047 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:49:00.618349 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618331 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d5de92-3fae-46c6-91dc-44c9d0404700" containerName="registry" Jul 24 22:49:00.618349 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618346 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d5de92-3fae-46c6-91dc-44c9d0404700" containerName="registry" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618359 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-metric" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618366 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-metric" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618374 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb69350f-3924-4b6a-a52d-bcf40d589078" containerName="console" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618380 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb69350f-3924-4b6a-a52d-bcf40d589078" containerName="console" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618406 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618414 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618426 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="prom-label-proxy" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618434 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="prom-label-proxy" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618443 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="init-config-reloader" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618451 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="init-config-reloader" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618464 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="config-reloader" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618472 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="config-reloader" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618484 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-web" Jul 24 22:49:00.618493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618492 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-web" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618509 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="alertmanager" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618517 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="alertmanager" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618587 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d5de92-3fae-46c6-91dc-44c9d0404700" containerName="registry" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618597 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-web" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618606 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="prom-label-proxy" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618616 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="config-reloader" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618629 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb69350f-3924-4b6a-a52d-bcf40d589078" containerName="console" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618637 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618644 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="alertmanager" Jul 24 22:49:00.618966 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.618650 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" containerName="kube-rbac-proxy-metric" Jul 24 22:49:00.623963 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.623945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.626500 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626479 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 22:49:00.626614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626479 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-rgvjc\"" Jul 24 22:49:00.626614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626564 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 22:49:00.626614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626574 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 22:49:00.626796 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626671 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 22:49:00.626936 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626920 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 22:49:00.627024 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.626925 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 22:49:00.627079 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.627056 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 22:49:00.627138 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.627076 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 22:49:00.631664 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.631645 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 22:49:00.636572 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.636551 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:49:00.643481 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643459 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643582 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643582 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643507 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643582 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643524 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqrk\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-kube-api-access-5nqrk\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643710 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643636 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643710 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643677 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643809 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643703 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643809 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643765 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643914 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643815 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643914 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643858 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.643914 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643888 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.644052 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643912 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.644052 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.643983 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.744733 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744642 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.744733 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744684 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.744733 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744706 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.744733 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744721 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744765 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744805 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744830 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745054 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.744863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745801 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.745773 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.745973 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.745948 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746122 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.745790 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqrk\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-kube-api-access-5nqrk\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746122 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.746097 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.746145 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.746182 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.746227 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.746458 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.746434 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.747863 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.747835 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-web-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.747969 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.747868 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-out\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.747969 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.747906 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.748131 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.748108 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.748715 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.748690 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.748995 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.748971 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.749063 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.748992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.749063 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.749014 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.749369 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.749349 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.753298 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.753279 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqrk\" (UniqueName: \"kubernetes.io/projected/8fa88a77-56e7-4e6b-826c-1c33a6ea31c7-kube-api-access-5nqrk\") pod \"alertmanager-main-0\" (UID: \"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:00.933982 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:00.933945 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 22:49:01.075730 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:01.074923 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 22:49:01.079496 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:49:01.079463 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa88a77_56e7_4e6b_826c_1c33a6ea31c7.slice/crio-990989502b2ff380acf5414033e4b604147bd8e96ece62797b1be62aa5f9b2f0 WatchSource:0}: Error finding container 990989502b2ff380acf5414033e4b604147bd8e96ece62797b1be62aa5f9b2f0: Status 404 returned error can't find the container with id 990989502b2ff380acf5414033e4b604147bd8e96ece62797b1be62aa5f9b2f0 Jul 24 22:49:01.559922 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:01.559886 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ecd22c9-6bd2-48bc-bf06-2197be14b50a" path="/var/lib/kubelet/pods/5ecd22c9-6bd2-48bc-bf06-2197be14b50a/volumes" Jul 24 22:49:01.567046 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:01.567015 2571 generic.go:356] "Generic (PLEG): container finished" podID="8fa88a77-56e7-4e6b-826c-1c33a6ea31c7" containerID="424ce4f6ec2ab28dce528c31ef96012fb5e3a587a21b9289ad7570df9173ce76" exitCode=0 Jul 24 22:49:01.567404 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:01.567078 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerDied","Data":"424ce4f6ec2ab28dce528c31ef96012fb5e3a587a21b9289ad7570df9173ce76"} Jul 24 22:49:01.567404 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:01.567128 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"990989502b2ff380acf5414033e4b604147bd8e96ece62797b1be62aa5f9b2f0"} Jul 24 22:49:02.577254 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577217 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"295c5f07ef80fb4f6d8299f413b416878dec366d3b1f625a3beac7fb5e7c4904"} Jul 24 22:49:02.577746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577297 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"702321a86557257e388f9cbbca2bf00bdd063d3c6dc7ff8f8ed393d4b94bd4d3"} Jul 24 22:49:02.577746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577341 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"537cf91b6651c1f06ce0bcebd9c0baf7dc43f61c17f19e9124b05eeaacd4d84c"} Jul 24 22:49:02.577746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577382 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"3117a51a2e13b66aec6326ddb30129bd0ece588a98f5e752a28bdb1a0c971230"} Jul 24 22:49:02.577746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577455 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"d80c91f576ba540c51d0001881f56e537a76f113527585cbc3cc0f5aca712f9d"} Jul 24 22:49:02.577746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.577493 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"8fa88a77-56e7-4e6b-826c-1c33a6ea31c7","Type":"ContainerStarted","Data":"624545fee345601b69df19c6f10c1711e1c83e0c9632d81426a210c797c9c99f"} Jul 24 22:49:02.602289 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.602204 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.586840336 podStartE2EDuration="2.60218442s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.344084ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:49:02.600495626 +0000 UTC m=+171.529834839" watchObservedRunningTime="2026-07-24 22:49:02.60218442 +0000 UTC m=+171.531523633" Jul 24 22:49:02.635680 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.635649 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:02.636063 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636028 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="prometheus" containerID="cri-o://0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" gracePeriod=600 Jul 24 22:49:02.636290 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636058 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="config-reloader" containerID="cri-o://386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" gracePeriod=600 Jul 24 22:49:02.636382 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636058 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-web" containerID="cri-o://a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" gracePeriod=600 Jul 24 22:49:02.636469 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636058 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="thanos-sidecar" containerID="cri-o://b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" gracePeriod=600 Jul 24 22:49:02.636519 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636081 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-thanos" containerID="cri-o://20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" gracePeriod=600 Jul 24 22:49:02.636519 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.636127 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy" containerID="cri-o://8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" gracePeriod=600 Jul 24 22:49:02.886509 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.886486 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:02.965531 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965500 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.965531 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965538 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965559 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965590 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965617 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965650 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965692 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965750 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965773 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965799 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svptp\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965823 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965851 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965889 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965924 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965949 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965964 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.965979 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966039 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.966134 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966061 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966091 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966122 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966167 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2\") pod \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\" (UID: \"4631710d-7e8c-4dd2-ad18-4aad6c9faa15\") " Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966180 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966262 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966348 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966536 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966600 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966618 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-metrics-client-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966636 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966652 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966667 2571 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967034 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.966681 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:02.967668 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.967292 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:49:02.969586 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.969545 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:49:02.970109 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970044 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config" (OuterVolumeSpecName: "config") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.970109 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970043 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out" (OuterVolumeSpecName: "config-out") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:49:02.970109 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970082 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.970109 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970101 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.970109 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970114 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:49:02.970447 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970166 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.970560 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970533 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp" (OuterVolumeSpecName: "kube-api-access-svptp") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "kube-api-access-svptp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:49:02.970661 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970635 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.970892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.970863 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.971249 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.971228 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.971445 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.971429 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:02.979870 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:02.979841 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config" (OuterVolumeSpecName: "web-config") pod "4631710d-7e8c-4dd2-ad18-4aad6c9faa15" (UID: "4631710d-7e8c-4dd2-ad18-4aad6c9faa15"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:49:03.068008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.067970 2571 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-metrics-client-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068003 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068013 2571 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-web-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068022 2571 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config-out\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068031 2571 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-kube-rbac-proxy\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068040 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svptp\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-kube-api-access-svptp\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068050 2571 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068058 2571 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-tls-assets\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068067 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068077 2571 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-grpc-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068086 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068094 2571 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-config\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068101 2571 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-prometheus-k8s-db\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.068194 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.068112 2571 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4631710d-7e8c-4dd2-ad18-4aad6c9faa15-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:49:03.584516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584483 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" exitCode=0 Jul 24 22:49:03.584516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584509 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" exitCode=0 Jul 24 22:49:03.584516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584516 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" exitCode=0 Jul 24 22:49:03.584516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584521 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" exitCode=0 Jul 24 22:49:03.584516 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584526 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" exitCode=0 Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584532 2571 generic.go:356] "Generic (PLEG): container finished" podID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" exitCode=0 Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584561 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584630 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584573 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584658 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584682 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584709 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584727 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584734 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} Jul 24 22:49:03.585037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.584827 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4631710d-7e8c-4dd2-ad18-4aad6c9faa15","Type":"ContainerDied","Data":"b1a989df30ab735fd6712bed04e541b5a495b9706bcc007fd9c1ea4ca8f4ec07"} Jul 24 22:49:03.593178 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.593159 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.600182 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.600164 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.602880 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.602856 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:03.608446 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.608427 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.609018 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.608996 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:03.615089 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.615069 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.621998 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.621969 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.629296 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.629269 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.634220 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634200 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:03.634535 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634522 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-thanos" Jul 24 22:49:03.634576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634537 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-thanos" Jul 24 22:49:03.634576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634552 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="prometheus" Jul 24 22:49:03.634576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634561 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="prometheus" Jul 24 22:49:03.634576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634567 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="thanos-sidecar" Jul 24 22:49:03.634576 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634573 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="thanos-sidecar" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634583 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="config-reloader" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634588 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="config-reloader" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634598 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="init-config-reloader" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634604 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="init-config-reloader" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634612 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-web" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634617 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-web" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634624 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634632 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634688 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634695 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-thanos" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634704 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="kube-rbac-proxy-web" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634715 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="prometheus" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634721 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="thanos-sidecar" Jul 24 22:49:03.634724 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.634727 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" containerName="config-reloader" Jul 24 22:49:03.636204 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636185 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.636455 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.636439 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.636512 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636463 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.636512 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636480 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.636677 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.636662 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.636741 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636680 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.636741 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636691 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.636875 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.636861 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.636917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636878 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.636917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.636890 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.637100 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.637071 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.637166 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637108 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.637166 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637130 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.637382 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.637365 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.637454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637388 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.637454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637423 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.637640 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.637624 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.637752 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637646 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.637752 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637664 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.637910 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:49:03.637895 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.637948 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637916 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.637948 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.637930 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.638120 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638101 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.638162 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638122 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.638304 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638286 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.638372 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638306 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.638529 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638513 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.638599 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638530 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.638756 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638738 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.638813 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638757 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.638950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638930 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.638993 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.638952 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.639173 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639149 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.639245 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639174 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.639345 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639325 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.639460 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639406 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.639460 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639426 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.639683 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639662 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.639757 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639685 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.639937 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639911 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.639937 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.639936 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.640214 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640174 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.640214 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640200 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.640464 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640444 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.640533 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640465 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.640698 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640673 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.640698 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640698 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.640914 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640897 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.640960 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.640914 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.641168 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641145 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.641215 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641171 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.641416 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641378 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 22:49:03.641493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641386 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 22:49:03.641493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641381 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.641493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641465 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 22:49:03.641493 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641466 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.641715 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641669 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 22:49:03.641821 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641805 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 22:49:03.641898 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641809 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 22:49:03.641898 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641802 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.641898 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641863 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7edg8tqfbn0f5\"" Jul 24 22:49:03.641898 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641881 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 22:49:03.641898 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.641865 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.642159 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642142 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 22:49:03.642218 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642142 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 22:49:03.642218 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642181 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.642218 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642200 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.642370 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642261 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 22:49:03.642488 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642468 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.642567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642491 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.642567 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642555 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2snj6\"" Jul 24 22:49:03.642664 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642585 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 22:49:03.642718 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642678 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 22:49:03.642833 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642786 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.642889 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.642835 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.643123 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643094 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.643188 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643126 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.643453 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643420 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.643534 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643455 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.643799 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643753 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.643799 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.643780 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.644050 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644025 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.644050 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644045 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.644277 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644256 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.644277 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644276 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.644547 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644524 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.644627 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644551 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.644848 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644826 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.644848 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.644847 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.645151 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645128 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.645234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645153 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.645234 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645167 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 22:49:03.645491 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645445 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.645581 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645494 2571 scope.go:117] "RemoveContainer" containerID="20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404" Jul 24 22:49:03.645834 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645814 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404"} err="failed to get container status \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": rpc error: code = NotFound desc = could not find container \"20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404\": container with ID starting with 20652cd6b42a42a51f1333da5f2ed91e0a9898cd7debbd5205e99963d9e54404 not found: ID does not exist" Jul 24 22:49:03.645913 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.645836 2571 scope.go:117] "RemoveContainer" containerID="8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf" Jul 24 22:49:03.646095 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646076 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf"} err="failed to get container status \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": rpc error: code = NotFound desc = could not find container \"8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf\": container with ID starting with 8b3635331f815f3d928174def06835ea4f48d47d4d05e4d8bb6c4b0f7013cdaf not found: ID does not exist" Jul 24 22:49:03.646173 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646097 2571 scope.go:117] "RemoveContainer" containerID="a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3" Jul 24 22:49:03.646456 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646434 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3"} err="failed to get container status \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": rpc error: code = NotFound desc = could not find container \"a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3\": container with ID starting with a116005f8a10a343b698c21c77afb63f25b477b0987a1918817799340afc01b3 not found: ID does not exist" Jul 24 22:49:03.646545 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646456 2571 scope.go:117] "RemoveContainer" containerID="b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b" Jul 24 22:49:03.646766 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646741 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b"} err="failed to get container status \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": rpc error: code = NotFound desc = could not find container \"b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b\": container with ID starting with b0e6052302e645ba3c4b0af4aa01d7d98c778d4a3fef2d48531d1a09ba87a88b not found: ID does not exist" Jul 24 22:49:03.646895 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.646771 2571 scope.go:117] "RemoveContainer" containerID="386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c" Jul 24 22:49:03.647107 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647087 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c"} err="failed to get container status \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": rpc error: code = NotFound desc = could not find container \"386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c\": container with ID starting with 386297b17260adf66c592f417b70c978d5864184df7810533288b3e6d2a2ae9c not found: ID does not exist" Jul 24 22:49:03.647191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647108 2571 scope.go:117] "RemoveContainer" containerID="0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0" Jul 24 22:49:03.647264 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647240 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 22:49:03.647438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647418 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0"} err="failed to get container status \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": rpc error: code = NotFound desc = could not find container \"0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0\": container with ID starting with 0585d4b1312dde0cf169d4d2fa351e1d83b104fd8ab4dabe8a28ceba624a15f0 not found: ID does not exist" Jul 24 22:49:03.647438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647438 2571 scope.go:117] "RemoveContainer" containerID="225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703" Jul 24 22:49:03.647697 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.647672 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703"} err="failed to get container status \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": rpc error: code = NotFound desc = could not find container \"225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703\": container with ID starting with 225df1c3994929caf1e8f4dd8048d6db6c4029f12663ac626b1fe59738774703 not found: ID does not exist" Jul 24 22:49:03.652191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.652144 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:03.672852 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.672827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.672873 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.672925 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jw97\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-kube-api-access-6jw97\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.672970 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673008 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.672994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673022 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673192 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673179 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673222 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673214 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673270 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673250 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673328 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673314 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673363 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673339 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673359 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673462 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673429 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673462 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673448 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673554 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673466 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673554 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673526 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673654 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673566 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673654 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673601 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673757 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.673757 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.673715 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774278 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774233 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774278 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774276 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774300 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774329 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774355 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774377 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774455 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774486 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774565 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774534 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774576 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774598 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jw97\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-kube-api-access-6jw97\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774629 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774652 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774682 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774709 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774731 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774756 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774801 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774834 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.774894 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.774863 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.775438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.775153 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.775438 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.775251 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.775844 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.775584 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.775844 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.775684 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.778071 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.778044 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.778633 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.778607 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.778738 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.778703 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.778807 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.778759 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0a57d006-50c6-476d-a383-3e0a6081db7d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.778954 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.778930 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-web-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779065 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779040 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779492 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779470 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0a57d006-50c6-476d-a383-3e0a6081db7d-config-out\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779600 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779475 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779832 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779810 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-config\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779933 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779911 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.779988 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.779945 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.780044 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.780010 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.780559 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.780540 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.781140 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.780924 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.786471 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.781701 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0a57d006-50c6-476d-a383-3e0a6081db7d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.787846 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.787816 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jw97\" (UniqueName: \"kubernetes.io/projected/0a57d006-50c6-476d-a383-3e0a6081db7d-kube-api-access-6jw97\") pod \"prometheus-k8s-0\" (UID: \"0a57d006-50c6-476d-a383-3e0a6081db7d\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:03.951455 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:03.951359 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:49:04.089944 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:04.089907 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 22:49:04.095642 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:49:04.095599 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a57d006_50c6_476d_a383_3e0a6081db7d.slice/crio-43b7530b453b9e2d18c5fa4ce2a247d00a2d1af0dc288f951b62b9a70d9100a7 WatchSource:0}: Error finding container 43b7530b453b9e2d18c5fa4ce2a247d00a2d1af0dc288f951b62b9a70d9100a7: Status 404 returned error can't find the container with id 43b7530b453b9e2d18c5fa4ce2a247d00a2d1af0dc288f951b62b9a70d9100a7 Jul 24 22:49:04.591050 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:04.590967 2571 generic.go:356] "Generic (PLEG): container finished" podID="0a57d006-50c6-476d-a383-3e0a6081db7d" containerID="67f6d6f286e2bb81572e48686af15dd8eca921b7e31bc02ee01cba8f28028a77" exitCode=0 Jul 24 22:49:04.591050 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:04.591036 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerDied","Data":"67f6d6f286e2bb81572e48686af15dd8eca921b7e31bc02ee01cba8f28028a77"} Jul 24 22:49:04.591491 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:04.591089 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"43b7530b453b9e2d18c5fa4ce2a247d00a2d1af0dc288f951b62b9a70d9100a7"} Jul 24 22:49:05.560308 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.560275 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4631710d-7e8c-4dd2-ad18-4aad6c9faa15" path="/var/lib/kubelet/pods/4631710d-7e8c-4dd2-ad18-4aad6c9faa15/volumes" Jul 24 22:49:05.599564 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599529 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"d87167a55e92d268b236d8d600d86baca3c34d5ad1055e7e9108ec95888867d2"} Jul 24 22:49:05.599950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599593 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"8a06aec14e3b2ab0a1bff4cee4309614d82dcf0ab52b10a89d41ae2122406623"} Jul 24 22:49:05.599950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599620 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"2d63f829c87dc2d8622be24f2cbd1dda17d88cfa83716a2be17831554dfda0c9"} Jul 24 22:49:05.599950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599644 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"782b224a135d126152b0462d903b6bc398f14bde7c55f1d34e1063504f220f8c"} Jul 24 22:49:05.599950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599672 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"6135923e6ec87666a1bfa6cfb63009dafa3c3754e80c62950c5de1ba6b5a6b39"} Jul 24 22:49:05.599950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.599697 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0a57d006-50c6-476d-a383-3e0a6081db7d","Type":"ContainerStarted","Data":"df12e4e81cc6ef5cf28acb54bf7b11d644d0d82c587aaf80eb291cef8ceff06d"} Jul 24 22:49:05.627988 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:05.627937 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.610891376 podStartE2EDuration="2.627919865s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.028489ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:49:05.625388084 +0000 UTC m=+174.554727306" watchObservedRunningTime="2026-07-24 22:49:05.627919865 +0000 UTC m=+174.557259076" Jul 24 22:49:08.952037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:49:08.951991 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:50:03.952612 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:50:03.952560 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:50:03.968443 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:50:03.968414 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:50:04.863602 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:50:04.863570 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 22:51:11.451881 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:51:11.451845 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:51:11.452415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:51:11.452012 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:51:11.457351 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:51:11.457327 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:51:11.457641 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:51:11.457619 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:51:11.462676 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:51:11.462661 2571 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 22:52:24.165859 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.165772 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj"] Jul 24 22:52:24.169512 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.169487 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.171838 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.171811 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2r4tk\"" Jul 24 22:52:24.171968 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.171865 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 22:52:24.172155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.172136 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 22:52:24.172276 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.172258 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 22:52:24.172639 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.172615 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 22:52:24.188466 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.188431 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj"] Jul 24 22:52:24.216998 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.216957 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.216998 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.216993 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdjx\" (UniqueName: \"kubernetes.io/projected/1066e233-706a-4088-a4fe-fbd5a7aac431-kube-api-access-dgdjx\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.217223 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.217020 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.217223 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.217053 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.217223 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.217114 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.318515 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.318479 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.318727 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.318521 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdjx\" (UniqueName: \"kubernetes.io/projected/1066e233-706a-4088-a4fe-fbd5a7aac431-kube-api-access-dgdjx\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.318801 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.318719 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.318801 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.318767 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.318912 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.318824 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.319181 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.319152 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-manifests\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.320909 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.320886 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1066e233-706a-4088-a4fe-fbd5a7aac431-tmp\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.320990 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.320917 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.321031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.320991 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1066e233-706a-4088-a4fe-fbd5a7aac431-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.326091 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.326072 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdjx\" (UniqueName: \"kubernetes.io/projected/1066e233-706a-4088-a4fe-fbd5a7aac431-kube-api-access-dgdjx\") pod \"opendatahub-operator-controller-manager-69d99768bd-9fbmj\" (UID: \"1066e233-706a-4088-a4fe-fbd5a7aac431\") " pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.482963 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.482874 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:24.612917 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.612883 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj"] Jul 24 22:52:24.616430 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:52:24.616388 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1066e233_706a_4088_a4fe_fbd5a7aac431.slice/crio-4e757b4a112765f31006ad1c23356b4a5f03ba3731e711afcedf63b38121610d WatchSource:0}: Error finding container 4e757b4a112765f31006ad1c23356b4a5f03ba3731e711afcedf63b38121610d: Status 404 returned error can't find the container with id 4e757b4a112765f31006ad1c23356b4a5f03ba3731e711afcedf63b38121610d Jul 24 22:52:24.618229 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:24.618209 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:52:25.413339 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:25.413303 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" event={"ID":"1066e233-706a-4088-a4fe-fbd5a7aac431","Type":"ContainerStarted","Data":"4e757b4a112765f31006ad1c23356b4a5f03ba3731e711afcedf63b38121610d"} Jul 24 22:52:28.428808 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:28.428767 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" event={"ID":"1066e233-706a-4088-a4fe-fbd5a7aac431","Type":"ContainerStarted","Data":"d3df20a07d97bcf238ee7157f679c22e7f37db88f36da469474303a93f3d316d"} Jul 24 22:52:29.434469 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:29.434436 2571 generic.go:356] "Generic (PLEG): container finished" podID="1066e233-706a-4088-a4fe-fbd5a7aac431" containerID="d3df20a07d97bcf238ee7157f679c22e7f37db88f36da469474303a93f3d316d" exitCode=0 Jul 24 22:52:29.434469 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:29.434474 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" event={"ID":"1066e233-706a-4088-a4fe-fbd5a7aac431","Type":"ContainerDied","Data":"d3df20a07d97bcf238ee7157f679c22e7f37db88f36da469474303a93f3d316d"} Jul 24 22:52:30.440498 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:30.440456 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" event={"ID":"1066e233-706a-4088-a4fe-fbd5a7aac431","Type":"ContainerStarted","Data":"5ce6d1ef626ff5b95fa1d53dba5281a7f9248e3c17cdbaffcf4f63dcc269df1c"} Jul 24 22:52:30.440955 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:30.440561 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:30.460593 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:30.460536 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" podStartSLOduration=2.111868636 podStartE2EDuration="6.460521074s" totalImagesPullingTime="3.357060846s" totalInitContainerRuntime="991.591592ms" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:30.458817761 +0000 UTC m=+379.388156974" watchObservedRunningTime="2026-07-24 22:52:30.460521074 +0000 UTC m=+379.389860359" Jul 24 22:52:41.448221 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:41.448182 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69d99768bd-9fbmj" Jul 24 22:52:51.123216 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.123177 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb"] Jul 24 22:52:51.129825 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.129806 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.133216 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133198 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 22:52:51.133216 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133208 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 22:52:51.133364 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133215 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 22:52:51.133541 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133523 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-stbf4\"" Jul 24 22:52:51.133657 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133640 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 22:52:51.133746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.133730 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 22:52:51.143687 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.143666 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb"] Jul 24 22:52:51.257120 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.257088 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-metrics-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.257303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.257136 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvgb\" (UniqueName: \"kubernetes.io/projected/c178989e-b63b-4bfb-a6d2-06c8bc14943b-kube-api-access-khvgb\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.257303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.257199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.257303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.257252 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c178989e-b63b-4bfb-a6d2-06c8bc14943b-manager-config\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.358543 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.358504 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-metrics-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.358701 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.358551 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvgb\" (UniqueName: \"kubernetes.io/projected/c178989e-b63b-4bfb-a6d2-06c8bc14943b-kube-api-access-khvgb\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.358701 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.358600 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.358701 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.358639 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c178989e-b63b-4bfb-a6d2-06c8bc14943b-manager-config\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.359220 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.359200 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c178989e-b63b-4bfb-a6d2-06c8bc14943b-manager-config\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.360975 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.360951 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-metrics-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.361141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.361118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c178989e-b63b-4bfb-a6d2-06c8bc14943b-cert\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.366816 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.366792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvgb\" (UniqueName: \"kubernetes.io/projected/c178989e-b63b-4bfb-a6d2-06c8bc14943b-kube-api-access-khvgb\") pod \"lws-controller-manager-86cd477748-6cxbb\" (UID: \"c178989e-b63b-4bfb-a6d2-06c8bc14943b\") " pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.439258 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.439166 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:51.579286 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:51.579213 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb"] Jul 24 22:52:51.581886 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:52:51.581846 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc178989e_b63b_4bfb_a6d2_06c8bc14943b.slice/crio-a8a4abd3e977ad7b4b8b7fc3823915a0ff489114931aac47fb3ea5bb7b4aa2f1 WatchSource:0}: Error finding container a8a4abd3e977ad7b4b8b7fc3823915a0ff489114931aac47fb3ea5bb7b4aa2f1: Status 404 returned error can't find the container with id a8a4abd3e977ad7b4b8b7fc3823915a0ff489114931aac47fb3ea5bb7b4aa2f1 Jul 24 22:52:52.540011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:52.539968 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" event={"ID":"c178989e-b63b-4bfb-a6d2-06c8bc14943b","Type":"ContainerStarted","Data":"a8a4abd3e977ad7b4b8b7fc3823915a0ff489114931aac47fb3ea5bb7b4aa2f1"} Jul 24 22:52:54.157357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.157321 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t"] Jul 24 22:52:54.160655 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.159916 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.162804 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.162732 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 22:52:54.162804 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.162786 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 22:52:54.163037 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.162862 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9fprg\"" Jul 24 22:52:54.184004 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.183967 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t"] Jul 24 22:52:54.288592 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.288551 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cff92f8-a35f-411a-8cca-4155434c3ed9-tls-certs\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.288777 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.288620 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cff92f8-a35f-411a-8cca-4155434c3ed9-tmp\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.288777 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.288665 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tcc\" (UniqueName: \"kubernetes.io/projected/9cff92f8-a35f-411a-8cca-4155434c3ed9-kube-api-access-97tcc\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.390187 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.390092 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cff92f8-a35f-411a-8cca-4155434c3ed9-tmp\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.390187 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.390134 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tcc\" (UniqueName: \"kubernetes.io/projected/9cff92f8-a35f-411a-8cca-4155434c3ed9-kube-api-access-97tcc\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.390448 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.390232 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cff92f8-a35f-411a-8cca-4155434c3ed9-tls-certs\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.392551 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.392524 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9cff92f8-a35f-411a-8cca-4155434c3ed9-tmp\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.392666 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.392649 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9cff92f8-a35f-411a-8cca-4155434c3ed9-tls-certs\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.398284 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.398263 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tcc\" (UniqueName: \"kubernetes.io/projected/9cff92f8-a35f-411a-8cca-4155434c3ed9-kube-api-access-97tcc\") pod \"kube-auth-proxy-849489fbdc-f7q7t\" (UID: \"9cff92f8-a35f-411a-8cca-4155434c3ed9\") " pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.470250 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.470210 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" Jul 24 22:52:54.551787 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.551742 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" event={"ID":"c178989e-b63b-4bfb-a6d2-06c8bc14943b","Type":"ContainerStarted","Data":"a3e7908513efc824a108b3b55ca19526cde7cea06ad957cc182f8e7155525910"} Jul 24 22:52:54.551962 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.551864 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:52:54.571655 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.571101 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" podStartSLOduration=1.080390551 podStartE2EDuration="3.571035197s" totalImagesPullingTime="2.490644646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:54.570379607 +0000 UTC m=+403.499718844" watchObservedRunningTime="2026-07-24 22:52:54.571035197 +0000 UTC m=+403.500374412" Jul 24 22:52:54.605129 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:54.605096 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t"] Jul 24 22:52:54.608776 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:52:54.608748 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cff92f8_a35f_411a_8cca_4155434c3ed9.slice/crio-197195302e97c9527a5fcffc36360c8c39e5bc98f77ad9c06c964d8a77b39b70 WatchSource:0}: Error finding container 197195302e97c9527a5fcffc36360c8c39e5bc98f77ad9c06c964d8a77b39b70: Status 404 returned error can't find the container with id 197195302e97c9527a5fcffc36360c8c39e5bc98f77ad9c06c964d8a77b39b70 Jul 24 22:52:55.560338 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:55.560298 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" event={"ID":"9cff92f8-a35f-411a-8cca-4155434c3ed9","Type":"ContainerStarted","Data":"197195302e97c9527a5fcffc36360c8c39e5bc98f77ad9c06c964d8a77b39b70"} Jul 24 22:52:57.571504 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:57.571460 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" event={"ID":"9cff92f8-a35f-411a-8cca-4155434c3ed9","Type":"ContainerStarted","Data":"5103ab7a740e6db8d41b8c343abd07be55bf088fdf967e800f2a426bbe1ff055"} Jul 24 22:52:57.587012 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:52:57.586956 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-849489fbdc-f7q7t" podStartSLOduration=1.239704953 podStartE2EDuration="3.586939351s" totalImagesPullingTime="2.347234398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:52:57.585638133 +0000 UTC m=+406.514977338" watchObservedRunningTime="2026-07-24 22:52:57.586939351 +0000 UTC m=+406.516278563" Jul 24 22:53:05.561546 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:53:05.561509 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-86cd477748-6cxbb" Jul 24 22:54:36.675680 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.675642 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg"] Jul 24 22:54:36.679006 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.678986 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.680784 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.680760 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 24 22:54:36.681002 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.680987 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pqjc8\"" Jul 24 22:54:36.681088 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.681001 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 22:54:36.681088 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.681001 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 24 22:54:36.681237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.681204 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 22:54:36.689025 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.689002 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg"] Jul 24 22:54:36.797607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.797564 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e83a82-4e36-4721-9f09-de0099368511-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.797607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.797612 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnqn4\" (UniqueName: \"kubernetes.io/projected/b4e83a82-4e36-4721-9f09-de0099368511-kube-api-access-jnqn4\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.797812 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.797691 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.899070 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.899024 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e83a82-4e36-4721-9f09-de0099368511-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.899070 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.899073 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnqn4\" (UniqueName: \"kubernetes.io/projected/b4e83a82-4e36-4721-9f09-de0099368511-kube-api-access-jnqn4\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.899299 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.899109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.899299 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:54:36.899209 2571 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 24 22:54:36.899299 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:54:36.899269 2571 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert podName:b4e83a82-4e36-4721-9f09-de0099368511 nodeName:}" failed. No retries permitted until 2026-07-24 22:54:37.399252649 +0000 UTC m=+506.328591840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-mjgkg" (UID: "b4e83a82-4e36-4721-9f09-de0099368511") : secret "plugin-serving-cert" not found Jul 24 22:54:36.899693 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.899669 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b4e83a82-4e36-4721-9f09-de0099368511-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:36.907332 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:36.907303 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnqn4\" (UniqueName: \"kubernetes.io/projected/b4e83a82-4e36-4721-9f09-de0099368511-kube-api-access-jnqn4\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:37.403477 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:37.403437 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:37.405914 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:37.405876 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b4e83a82-4e36-4721-9f09-de0099368511-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-mjgkg\" (UID: \"b4e83a82-4e36-4721-9f09-de0099368511\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:37.598854 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:37.598820 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" Jul 24 22:54:37.732746 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:37.732720 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg"] Jul 24 22:54:37.735043 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:54:37.735017 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e83a82_4e36_4721_9f09_de0099368511.slice/crio-cbb36959dfb9faae5044386bfe0b17aeeaffd2bf2d26f89e3c86b7ecf112ed81 WatchSource:0}: Error finding container cbb36959dfb9faae5044386bfe0b17aeeaffd2bf2d26f89e3c86b7ecf112ed81: Status 404 returned error can't find the container with id cbb36959dfb9faae5044386bfe0b17aeeaffd2bf2d26f89e3c86b7ecf112ed81 Jul 24 22:54:38.019985 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:38.019951 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" event={"ID":"b4e83a82-4e36-4721-9f09-de0099368511","Type":"ContainerStarted","Data":"cbb36959dfb9faae5044386bfe0b17aeeaffd2bf2d26f89e3c86b7ecf112ed81"} Jul 24 22:54:46.976530 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:46.976497 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:54:46.998135 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:46.998103 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:54:46.998298 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:46.998212 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.000641 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.000619 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-l9czn\"" Jul 24 22:54:47.102684 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.102647 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.102869 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.102723 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrwd\" (UniqueName: \"kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.204074 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.204030 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrwd\" (UniqueName: \"kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.204254 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.204181 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.204644 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.204622 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.212721 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.212688 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrwd\" (UniqueName: \"kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-dpw8x\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:47.314962 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:47.314925 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:54:59.016257 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:59.016218 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:54:59.018348 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:54:59.018322 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01647c6e_08b2_4bd9_b1ce_6487b16906a0.slice/crio-f8be7fe85ac89432677407e423ad8da24fea768940ed09dd1247e7af6d339c93 WatchSource:0}: Error finding container f8be7fe85ac89432677407e423ad8da24fea768940ed09dd1247e7af6d339c93: Status 404 returned error can't find the container with id f8be7fe85ac89432677407e423ad8da24fea768940ed09dd1247e7af6d339c93 Jul 24 22:54:59.129967 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:59.129880 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" event={"ID":"01647c6e-08b2-4bd9-b1ce-6487b16906a0","Type":"ContainerStarted","Data":"f8be7fe85ac89432677407e423ad8da24fea768940ed09dd1247e7af6d339c93"} Jul 24 22:54:59.131494 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:59.131461 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" event={"ID":"b4e83a82-4e36-4721-9f09-de0099368511","Type":"ContainerStarted","Data":"b10bb87730b6eb5cbed1eefcf8b94939e67ac2e92ee961b05fd996e9761a8e04"} Jul 24 22:54:59.156610 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:54:59.156557 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-mjgkg" podStartSLOduration=1.928616917 podStartE2EDuration="23.156533095s" totalImagesPullingTime="21.227916178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:54:59.155500031 +0000 UTC m=+528.084839244" watchObservedRunningTime="2026-07-24 22:54:59.156533095 +0000 UTC m=+528.085872305" Jul 24 22:55:05.164789 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:05.164747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" event={"ID":"01647c6e-08b2-4bd9-b1ce-6487b16906a0","Type":"ContainerStarted","Data":"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1"} Jul 24 22:55:05.165242 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:05.164834 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:55:05.189623 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:05.189572 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" podStartSLOduration=13.585895144 podStartE2EDuration="19.189557272s" totalImagesPullingTime="5.603662128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:05.187564448 +0000 UTC m=+534.116903672" watchObservedRunningTime="2026-07-24 22:55:05.189557272 +0000 UTC m=+534.118896482" Jul 24 22:55:16.172011 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:16.171980 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:55:28.486922 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.486883 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:55:28.487368 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.487072 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" podUID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" containerName="manager" containerID="cri-o://d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1" gracePeriod=10 Jul 24 22:55:28.747999 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.747939 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:55:28.898528 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.898488 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume\") pod \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " Jul 24 22:55:28.898528 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.898527 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmrwd\" (UniqueName: \"kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd\") pod \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\" (UID: \"01647c6e-08b2-4bd9-b1ce-6487b16906a0\") " Jul 24 22:55:28.898905 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.898877 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "01647c6e-08b2-4bd9-b1ce-6487b16906a0" (UID: "01647c6e-08b2-4bd9-b1ce-6487b16906a0"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 22:55:28.900704 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:28.900682 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd" (OuterVolumeSpecName: "kube-api-access-zmrwd") pod "01647c6e-08b2-4bd9-b1ce-6487b16906a0" (UID: "01647c6e-08b2-4bd9-b1ce-6487b16906a0"). InnerVolumeSpecName "kube-api-access-zmrwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:55:29.000275 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.000165 2571 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/01647c6e-08b2-4bd9-b1ce-6487b16906a0-extensions-socket-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:55:29.000275 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.000214 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrwd\" (UniqueName: \"kubernetes.io/projected/01647c6e-08b2-4bd9-b1ce-6487b16906a0-kube-api-access-zmrwd\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:55:29.273482 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.273445 2571 generic.go:356] "Generic (PLEG): container finished" podID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" containerID="d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1" exitCode=0 Jul 24 22:55:29.273651 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.273490 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" Jul 24 22:55:29.273651 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.273532 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" event={"ID":"01647c6e-08b2-4bd9-b1ce-6487b16906a0","Type":"ContainerDied","Data":"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1"} Jul 24 22:55:29.273651 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.273584 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x" event={"ID":"01647c6e-08b2-4bd9-b1ce-6487b16906a0","Type":"ContainerDied","Data":"f8be7fe85ac89432677407e423ad8da24fea768940ed09dd1247e7af6d339c93"} Jul 24 22:55:29.273651 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.273607 2571 scope.go:117] "RemoveContainer" containerID="d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1" Jul 24 22:55:29.283309 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.283283 2571 scope.go:117] "RemoveContainer" containerID="d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1" Jul 24 22:55:29.283602 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:55:29.283585 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1\": container with ID starting with d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1 not found: ID does not exist" containerID="d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1" Jul 24 22:55:29.283659 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.283609 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1"} err="failed to get container status \"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1\": rpc error: code = NotFound desc = could not find container \"d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1\": container with ID starting with d8995f05900cbc7300534c4f005f09dbddefb075a0daa2da65fb341b7aa12ad1 not found: ID does not exist" Jul 24 22:55:29.294171 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.294144 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:55:29.300307 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.300284 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-dpw8x"] Jul 24 22:55:29.560197 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:29.560119 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" path="/var/lib/kubelet/pods/01647c6e-08b2-4bd9-b1ce-6487b16906a0/volumes" Jul 24 22:55:34.551759 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.551719 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:34.552246 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.552211 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" containerName="manager" Jul 24 22:55:34.552246 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.552231 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" containerName="manager" Jul 24 22:55:34.552371 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.552355 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="01647c6e-08b2-4bd9-b1ce-6487b16906a0" containerName="manager" Jul 24 22:55:34.579173 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.579130 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:34.579331 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.579264 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.581084 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.581061 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 22:55:34.651988 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.651951 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:34.750523 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.750488 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.750523 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.750525 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxd5v\" (UniqueName: \"kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.851143 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.851061 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.851143 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.851098 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxd5v\" (UniqueName: \"kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.851761 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.851739 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.858586 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.858553 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxd5v\" (UniqueName: \"kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v\") pod \"limitador-limitador-64c94fd64f-qg28g\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:34.889615 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:34.889584 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:35.023664 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:35.023638 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:35.026248 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:55:35.026219 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2223773_6870_468f_949d_b7b22cec9d31.slice/crio-18698ddb26bf72f3eb5a189ab10b2e3f6d8d93eebf843329962bdbdbafb32c21 WatchSource:0}: Error finding container 18698ddb26bf72f3eb5a189ab10b2e3f6d8d93eebf843329962bdbdbafb32c21: Status 404 returned error can't find the container with id 18698ddb26bf72f3eb5a189ab10b2e3f6d8d93eebf843329962bdbdbafb32c21 Jul 24 22:55:35.305570 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:35.305532 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" event={"ID":"b2223773-6870-468f-949d-b7b22cec9d31","Type":"ContainerStarted","Data":"18698ddb26bf72f3eb5a189ab10b2e3f6d8d93eebf843329962bdbdbafb32c21"} Jul 24 22:55:38.321777 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:38.321740 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" event={"ID":"b2223773-6870-468f-949d-b7b22cec9d31","Type":"ContainerStarted","Data":"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447"} Jul 24 22:55:38.322206 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:38.321825 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:38.337571 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:38.337527 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" podStartSLOduration=1.165413879 podStartE2EDuration="4.337508769s" totalImagesPullingTime="3.17209489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:55:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:38.336582807 +0000 UTC m=+567.265922021" watchObservedRunningTime="2026-07-24 22:55:38.337508769 +0000 UTC m=+567.266847982" Jul 24 22:55:49.181340 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.181302 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:49.181828 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.181553 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" podUID="b2223773-6870-468f-949d-b7b22cec9d31" containerName="limitador" containerID="cri-o://fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447" gracePeriod=30 Jul 24 22:55:49.182213 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.182193 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:49.725978 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.725956 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:49.788351 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.788321 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file\") pod \"b2223773-6870-468f-949d-b7b22cec9d31\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " Jul 24 22:55:49.788351 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.788353 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxd5v\" (UniqueName: \"kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v\") pod \"b2223773-6870-468f-949d-b7b22cec9d31\" (UID: \"b2223773-6870-468f-949d-b7b22cec9d31\") " Jul 24 22:55:49.788728 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.788702 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file" (OuterVolumeSpecName: "config-file") pod "b2223773-6870-468f-949d-b7b22cec9d31" (UID: "b2223773-6870-468f-949d-b7b22cec9d31"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 22:55:49.790454 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.790434 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v" (OuterVolumeSpecName: "kube-api-access-kxd5v") pod "b2223773-6870-468f-949d-b7b22cec9d31" (UID: "b2223773-6870-468f-949d-b7b22cec9d31"). InnerVolumeSpecName "kube-api-access-kxd5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:55:49.889285 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.889258 2571 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b2223773-6870-468f-949d-b7b22cec9d31-config-file\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:55:49.889285 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:49.889282 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxd5v\" (UniqueName: \"kubernetes.io/projected/b2223773-6870-468f-949d-b7b22cec9d31-kube-api-access-kxd5v\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:55:50.378153 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.378116 2571 generic.go:356] "Generic (PLEG): container finished" podID="b2223773-6870-468f-949d-b7b22cec9d31" containerID="fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447" exitCode=0 Jul 24 22:55:50.378617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.378164 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" Jul 24 22:55:50.378617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.378204 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" event={"ID":"b2223773-6870-468f-949d-b7b22cec9d31","Type":"ContainerDied","Data":"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447"} Jul 24 22:55:50.378617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.378260 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-qg28g" event={"ID":"b2223773-6870-468f-949d-b7b22cec9d31","Type":"ContainerDied","Data":"18698ddb26bf72f3eb5a189ab10b2e3f6d8d93eebf843329962bdbdbafb32c21"} Jul 24 22:55:50.378617 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.378289 2571 scope.go:117] "RemoveContainer" containerID="fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447" Jul 24 22:55:50.387004 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.386982 2571 scope.go:117] "RemoveContainer" containerID="fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447" Jul 24 22:55:50.387262 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:55:50.387237 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447\": container with ID starting with fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447 not found: ID does not exist" containerID="fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447" Jul 24 22:55:50.387336 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.387271 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447"} err="failed to get container status \"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447\": rpc error: code = NotFound desc = could not find container \"fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447\": container with ID starting with fdd6c194d38eeaac700c3e6d1d1806d6508e3c0ed93b6720e835267cbf9d2447 not found: ID does not exist" Jul 24 22:55:50.397065 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.397033 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:50.399806 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:50.399786 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-qg28g"] Jul 24 22:55:51.091248 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.091217 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2"] Jul 24 22:55:51.091591 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.091573 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2223773-6870-468f-949d-b7b22cec9d31" containerName="limitador" Jul 24 22:55:51.091591 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.091584 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2223773-6870-468f-949d-b7b22cec9d31" containerName="limitador" Jul 24 22:55:51.091722 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.091648 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2223773-6870-468f-949d-b7b22cec9d31" containerName="limitador" Jul 24 22:55:51.095951 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.095929 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.097815 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.097792 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 22:55:51.097934 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.097834 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 22:55:51.098092 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.098075 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8nvkb\"" Jul 24 22:55:51.098155 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.098075 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 22:55:51.101970 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.101947 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2"] Jul 24 22:55:51.199581 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.199547 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-data\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.199794 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.199606 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjh4l\" (UniqueName: \"kubernetes.io/projected/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-kube-api-access-mjh4l\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.300945 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.300913 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-data\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.301069 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.300976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjh4l\" (UniqueName: \"kubernetes.io/projected/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-kube-api-access-mjh4l\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.301303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.301282 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-data\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.308705 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.308678 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjh4l\" (UniqueName: \"kubernetes.io/projected/1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc-kube-api-access-mjh4l\") pod \"postgres-69fb4594cf-lhlg2\" (UID: \"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.407274 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.407190 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:51.540307 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.540279 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2"] Jul 24 22:55:51.542776 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:55:51.542746 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ae7c29b_fa23_43fc_ae2a_cf3edf8499fc.slice/crio-63b9f462b7ad427d3fbaf8017ecb3d455674ed45859ec39aebe36ef807981dea WatchSource:0}: Error finding container 63b9f462b7ad427d3fbaf8017ecb3d455674ed45859ec39aebe36ef807981dea: Status 404 returned error can't find the container with id 63b9f462b7ad427d3fbaf8017ecb3d455674ed45859ec39aebe36ef807981dea Jul 24 22:55:51.559303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:51.559278 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2223773-6870-468f-949d-b7b22cec9d31" path="/var/lib/kubelet/pods/b2223773-6870-468f-949d-b7b22cec9d31/volumes" Jul 24 22:55:52.393861 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:52.393777 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" event={"ID":"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc","Type":"ContainerStarted","Data":"63b9f462b7ad427d3fbaf8017ecb3d455674ed45859ec39aebe36ef807981dea"} Jul 24 22:55:56.414638 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:56.414520 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" event={"ID":"1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc","Type":"ContainerStarted","Data":"e90adcb4e597df8934e102c9ef357d79b086b905eb12b1a57d46ef7e85c0ba83"} Jul 24 22:55:56.415029 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:56.414667 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:55:56.429945 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:55:56.429889 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" podStartSLOduration=0.841600291 podStartE2EDuration="5.429869104s" totalImagesPullingTime="4.588268813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:55:56.429290039 +0000 UTC m=+585.358629267" watchObservedRunningTime="2026-07-24 22:55:56.429869104 +0000 UTC m=+585.359208318" Jul 24 22:56:02.449189 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:02.449158 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-lhlg2" Jul 24 22:56:09.382607 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.382569 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:09.387349 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.387331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.389842 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.389795 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 22:56:09.390038 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.390017 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kgsz\"" Jul 24 22:56:09.390385 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.390369 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 22:56:09.400045 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.400014 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:09.476563 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.476531 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q8hg\" (UniqueName: \"kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.476720 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.476585 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.559531 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.559501 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 22:56:09.562348 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.562332 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.569722 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.569157 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 22:56:09.577243 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.577212 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.577378 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.577345 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q8hg\" (UniqueName: \"kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.579527 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.579507 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.585481 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.585461 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q8hg\" (UniqueName: \"kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg\") pod \"maas-controller-74c69bdc76-chthd\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.678658 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.678580 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.678658 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.678650 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7wgf\" (UniqueName: \"kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.697886 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.697866 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:09.779813 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.779778 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.779985 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.779862 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7wgf\" (UniqueName: \"kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.782575 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.782549 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.791094 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.791042 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7wgf\" (UniqueName: \"kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf\") pod \"maas-controller-9dbb4674-nfs8w\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:09.837449 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.836204 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:09.874386 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:09.874360 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:10.001241 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:10.001217 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 22:56:10.002952 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:56:10.002922 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560d79f6_7643_46e1_9456_44da9e8ec355.slice/crio-d63fa4a228071093f655479ed56b3e27629d5d8dc569177242cd5d0852f1c8d6 WatchSource:0}: Error finding container d63fa4a228071093f655479ed56b3e27629d5d8dc569177242cd5d0852f1c8d6: Status 404 returned error can't find the container with id d63fa4a228071093f655479ed56b3e27629d5d8dc569177242cd5d0852f1c8d6 Jul 24 22:56:10.483061 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:10.483001 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-nfs8w" event={"ID":"560d79f6-7643-46e1-9456-44da9e8ec355","Type":"ContainerStarted","Data":"d63fa4a228071093f655479ed56b3e27629d5d8dc569177242cd5d0852f1c8d6"} Jul 24 22:56:10.484575 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:10.484542 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c69bdc76-chthd" event={"ID":"347544c3-440a-4875-91f8-160434eebcdf","Type":"ContainerStarted","Data":"5d789b0a12ab0e468c69b1db965e503038830139dab35fe4290b0f396fae0fd5"} Jul 24 22:56:12.312490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.312467 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:56:12.312871 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.312519 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 22:56:12.318356 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.318338 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:56:12.318490 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.318343 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 22:56:12.354301 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.354284 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 22:56:12.495843 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.495792 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-nfs8w" event={"ID":"560d79f6-7643-46e1-9456-44da9e8ec355","Type":"ContainerStarted","Data":"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a"} Jul 24 22:56:12.496031 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.495906 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:12.497604 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.497577 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c69bdc76-chthd" event={"ID":"347544c3-440a-4875-91f8-160434eebcdf","Type":"ContainerStarted","Data":"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09"} Jul 24 22:56:12.497696 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.497627 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:12.517259 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.517197 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9dbb4674-nfs8w" podStartSLOduration=1.169358194 podStartE2EDuration="3.517181543s" totalImagesPullingTime="2.347823349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:12.514603334 +0000 UTC m=+601.443942546" watchObservedRunningTime="2026-07-24 22:56:12.517181543 +0000 UTC m=+601.446520754" Jul 24 22:56:12.537932 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:12.537823 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-74c69bdc76-chthd" podStartSLOduration=1.019556284 podStartE2EDuration="3.537806161s" totalImagesPullingTime="2.518249877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:12.537080048 +0000 UTC m=+601.466419273" watchObservedRunningTime="2026-07-24 22:56:12.537806161 +0000 UTC m=+601.467145374" Jul 24 22:56:23.506525 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.506494 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 22:56:23.506971 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.506603 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:23.561892 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.561864 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:23.562079 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.562026 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-74c69bdc76-chthd" podUID="347544c3-440a-4875-91f8-160434eebcdf" containerName="manager" containerID="cri-o://bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09" gracePeriod=30 Jul 24 22:56:23.809950 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.809919 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:23.813737 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.813714 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q8hg\" (UniqueName: \"kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg\") pod \"347544c3-440a-4875-91f8-160434eebcdf\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " Jul 24 22:56:23.813835 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.813766 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert\") pod \"347544c3-440a-4875-91f8-160434eebcdf\" (UID: \"347544c3-440a-4875-91f8-160434eebcdf\") " Jul 24 22:56:23.815808 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.815783 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg" (OuterVolumeSpecName: "kube-api-access-5q8hg") pod "347544c3-440a-4875-91f8-160434eebcdf" (UID: "347544c3-440a-4875-91f8-160434eebcdf"). InnerVolumeSpecName "kube-api-access-5q8hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 22:56:23.815891 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.815803 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "347544c3-440a-4875-91f8-160434eebcdf" (UID: "347544c3-440a-4875-91f8-160434eebcdf"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 22:56:23.914846 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.914810 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q8hg\" (UniqueName: \"kubernetes.io/projected/347544c3-440a-4875-91f8-160434eebcdf-kube-api-access-5q8hg\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:56:23.914846 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:23.914842 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/347544c3-440a-4875-91f8-160434eebcdf-webhook-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 22:56:24.557359 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.557319 2571 generic.go:356] "Generic (PLEG): container finished" podID="347544c3-440a-4875-91f8-160434eebcdf" containerID="bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09" exitCode=0 Jul 24 22:56:24.557774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.557364 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-74c69bdc76-chthd" Jul 24 22:56:24.557774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.557419 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c69bdc76-chthd" event={"ID":"347544c3-440a-4875-91f8-160434eebcdf","Type":"ContainerDied","Data":"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09"} Jul 24 22:56:24.557774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.557471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-74c69bdc76-chthd" event={"ID":"347544c3-440a-4875-91f8-160434eebcdf","Type":"ContainerDied","Data":"5d789b0a12ab0e468c69b1db965e503038830139dab35fe4290b0f396fae0fd5"} Jul 24 22:56:24.557774 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.557493 2571 scope.go:117] "RemoveContainer" containerID="bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09" Jul 24 22:56:24.566667 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.566488 2571 scope.go:117] "RemoveContainer" containerID="bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09" Jul 24 22:56:24.566820 ip-10-0-136-126 kubenswrapper[2571]: E0724 22:56:24.566801 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09\": container with ID starting with bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09 not found: ID does not exist" containerID="bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09" Jul 24 22:56:24.566871 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.566828 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09"} err="failed to get container status \"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09\": rpc error: code = NotFound desc = could not find container \"bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09\": container with ID starting with bbf6467206477596242e70c258d216dd95963f33ca4800db121b61c611dffd09 not found: ID does not exist" Jul 24 22:56:24.577835 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.577812 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:24.581487 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:24.581465 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-74c69bdc76-chthd"] Jul 24 22:56:25.565715 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:25.565677 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="347544c3-440a-4875-91f8-160434eebcdf" path="/var/lib/kubelet/pods/347544c3-440a-4875-91f8-160434eebcdf/volumes" Jul 24 22:56:41.178459 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.178362 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh"] Jul 24 22:56:41.179025 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.178953 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="347544c3-440a-4875-91f8-160434eebcdf" containerName="manager" Jul 24 22:56:41.179025 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.178975 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="347544c3-440a-4875-91f8-160434eebcdf" containerName="manager" Jul 24 22:56:41.179148 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.179091 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="347544c3-440a-4875-91f8-160434eebcdf" containerName="manager" Jul 24 22:56:41.182090 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.182067 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.184100 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.184075 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 24 22:56:41.184223 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.184110 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ftsjd\"" Jul 24 22:56:41.193921 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.193893 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh"] Jul 24 22:56:41.273929 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.273894 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwhf7\" (UniqueName: \"kubernetes.io/projected/903ff18a-64a5-4f10-99c7-1dbfb0961057-kube-api-access-gwhf7\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.274101 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.273954 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ff18a-64a5-4f10-99c7-1dbfb0961057-maas-api-tls\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.274101 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.274013 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ff18a-64a5-4f10-99c7-1dbfb0961057-openshift-service-ca\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.375072 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.375035 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwhf7\" (UniqueName: \"kubernetes.io/projected/903ff18a-64a5-4f10-99c7-1dbfb0961057-kube-api-access-gwhf7\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.375260 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.375099 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ff18a-64a5-4f10-99c7-1dbfb0961057-maas-api-tls\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.375260 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.375173 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ff18a-64a5-4f10-99c7-1dbfb0961057-openshift-service-ca\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.375847 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.375820 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/903ff18a-64a5-4f10-99c7-1dbfb0961057-openshift-service-ca\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.377613 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.377585 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/903ff18a-64a5-4f10-99c7-1dbfb0961057-maas-api-tls\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.383616 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.383594 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwhf7\" (UniqueName: \"kubernetes.io/projected/903ff18a-64a5-4f10-99c7-1dbfb0961057-kube-api-access-gwhf7\") pod \"maas-api-6d4c8b85cc-gb9wh\" (UID: \"903ff18a-64a5-4f10-99c7-1dbfb0961057\") " pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.492749 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.492670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:41.630841 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.630808 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh"] Jul 24 22:56:41.634227 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:56:41.634201 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903ff18a_64a5_4f10_99c7_1dbfb0961057.slice/crio-9cf58b7e9c719fac6e0ca43a2227edda3d8bc7ab8685576d67255a78f275aa45 WatchSource:0}: Error finding container 9cf58b7e9c719fac6e0ca43a2227edda3d8bc7ab8685576d67255a78f275aa45: Status 404 returned error can't find the container with id 9cf58b7e9c719fac6e0ca43a2227edda3d8bc7ab8685576d67255a78f275aa45 Jul 24 22:56:41.749769 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.749693 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4"] Jul 24 22:56:41.754181 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.754162 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.755935 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.755915 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 22:56:41.756040 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.755981 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-t6v89\"" Jul 24 22:56:41.763156 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.763132 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4"] Jul 24 22:56:41.773734 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.773708 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gczwg"] Jul 24 22:56:41.777100 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.777078 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.788170 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.788146 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gczwg"] Jul 24 22:56:41.879310 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.879274 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47034fdc-50f0-45b9-9ca7-354ebce2ec89-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.879508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.879334 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005b637c-a677-4214-889f-772b1f919ccf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.879508 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.879494 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q45c8\" (UniqueName: \"kubernetes.io/projected/005b637c-a677-4214-889f-772b1f919ccf-kube-api-access-q45c8\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.879614 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.879540 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwc7\" (UniqueName: \"kubernetes.io/projected/47034fdc-50f0-45b9-9ca7-354ebce2ec89-kube-api-access-wcwc7\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.980141 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980105 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q45c8\" (UniqueName: \"kubernetes.io/projected/005b637c-a677-4214-889f-772b1f919ccf-kube-api-access-q45c8\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.980315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980152 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwc7\" (UniqueName: \"kubernetes.io/projected/47034fdc-50f0-45b9-9ca7-354ebce2ec89-kube-api-access-wcwc7\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.980315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980177 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47034fdc-50f0-45b9-9ca7-354ebce2ec89-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.980315 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005b637c-a677-4214-889f-772b1f919ccf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.980858 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47034fdc-50f0-45b9-9ca7-354ebce2ec89-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:41.980988 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.980972 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/005b637c-a677-4214-889f-772b1f919ccf-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.990385 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.990355 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q45c8\" (UniqueName: \"kubernetes.io/projected/005b637c-a677-4214-889f-772b1f919ccf-kube-api-access-q45c8\") pod \"payload-processing-5cc6cd444b-gczwg\" (UID: \"005b637c-a677-4214-889f-772b1f919ccf\") " pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:41.991314 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:41.991292 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwc7\" (UniqueName: \"kubernetes.io/projected/47034fdc-50f0-45b9-9ca7-354ebce2ec89-kube-api-access-wcwc7\") pod \"payload-pre-processing-687dbb6cf6-cfnx4\" (UID: \"47034fdc-50f0-45b9-9ca7-354ebce2ec89\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:42.065704 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.065661 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:42.087125 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.087025 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:42.210924 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.210892 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4"] Jul 24 22:56:42.214321 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:56:42.214296 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47034fdc_50f0_45b9_9ca7_354ebce2ec89.slice/crio-0f36778b9f3ce8c9c9b957fc13033bcc5c68e7dbc6b2baa7a8eaeefc385b707e WatchSource:0}: Error finding container 0f36778b9f3ce8c9c9b957fc13033bcc5c68e7dbc6b2baa7a8eaeefc385b707e: Status 404 returned error can't find the container with id 0f36778b9f3ce8c9c9b957fc13033bcc5c68e7dbc6b2baa7a8eaeefc385b707e Jul 24 22:56:42.239651 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.239625 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-gczwg"] Jul 24 22:56:42.241353 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:56:42.241329 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005b637c_a677_4214_889f_772b1f919ccf.slice/crio-dc82e31df0e9028cc9501da3033317b32a858585523251de1db5af086de66f55 WatchSource:0}: Error finding container dc82e31df0e9028cc9501da3033317b32a858585523251de1db5af086de66f55: Status 404 returned error can't find the container with id dc82e31df0e9028cc9501da3033317b32a858585523251de1db5af086de66f55 Jul 24 22:56:42.646051 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.645977 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" event={"ID":"47034fdc-50f0-45b9-9ca7-354ebce2ec89","Type":"ContainerStarted","Data":"0f36778b9f3ce8c9c9b957fc13033bcc5c68e7dbc6b2baa7a8eaeefc385b707e"} Jul 24 22:56:42.647422 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.647364 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" event={"ID":"005b637c-a677-4214-889f-772b1f919ccf","Type":"ContainerStarted","Data":"dc82e31df0e9028cc9501da3033317b32a858585523251de1db5af086de66f55"} Jul 24 22:56:42.648827 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:42.648802 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" event={"ID":"903ff18a-64a5-4f10-99c7-1dbfb0961057","Type":"ContainerStarted","Data":"9cf58b7e9c719fac6e0ca43a2227edda3d8bc7ab8685576d67255a78f275aa45"} Jul 24 22:56:43.660049 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:43.659161 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" event={"ID":"903ff18a-64a5-4f10-99c7-1dbfb0961057","Type":"ContainerStarted","Data":"32cc1ea35f2da689e268f02a573b9dd756f256f70647d439fe3edb37f6805361"} Jul 24 22:56:43.660049 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:43.659418 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:43.675875 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:43.675775 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" podStartSLOduration=1.2117526299999999 podStartE2EDuration="2.675757443s" totalImagesPullingTime="1.464004813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:43.673936526 +0000 UTC m=+632.603275750" watchObservedRunningTime="2026-07-24 22:56:43.675757443 +0000 UTC m=+632.605096664" Jul 24 22:56:45.677857 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.677820 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" event={"ID":"47034fdc-50f0-45b9-9ca7-354ebce2ec89","Type":"ContainerStarted","Data":"9d75969ec2219bb49623c187b09a12e16b42d68d0f98d8813443a8e80ded06de"} Jul 24 22:56:45.678357 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.677907 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:56:45.679480 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.679456 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" event={"ID":"005b637c-a677-4214-889f-772b1f919ccf","Type":"ContainerStarted","Data":"4942c35499098cc6c57d384fd041ce170647ffe121d908d86be2d2f1d6a4ec32"} Jul 24 22:56:45.679571 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.679511 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:45.691433 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.691355 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" podStartSLOduration=2.166102285 podStartE2EDuration="4.691338502s" totalImagesPullingTime="2.525236217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:45.690441519 +0000 UTC m=+634.619780731" watchObservedRunningTime="2026-07-24 22:56:45.691338502 +0000 UTC m=+634.620677714" Jul 24 22:56:45.704304 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:45.704249 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" podStartSLOduration=2.2125996 podStartE2EDuration="4.70423097s" totalImagesPullingTime="2.49163137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:56:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:56:45.702507535 +0000 UTC m=+634.631846746" watchObservedRunningTime="2026-07-24 22:56:45.70423097 +0000 UTC m=+634.633570181" Jul 24 22:56:49.676496 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:49.676383 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d4c8b85cc-gb9wh" Jul 24 22:56:56.687139 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:56.687104 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-gczwg" Jul 24 22:56:56.687585 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:56:56.687176 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-cfnx4" Jul 24 22:57:40.750059 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.747219 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9"] Jul 24 22:57:40.752451 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.752424 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.755227 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.755202 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 24 22:57:40.755765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.755222 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 22:57:40.755765 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.755275 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4rw57\"" Jul 24 22:57:40.755960 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.755796 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 22:57:40.757375 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.757353 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9"] Jul 24 22:57:40.917428 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917375 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.917587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917440 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98fc85f-290e-42a0-88c9-70ec55212176-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.917587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917481 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.917587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917496 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.917587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917527 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptz9j\" (UniqueName: \"kubernetes.io/projected/b98fc85f-290e-42a0-88c9-70ec55212176-kube-api-access-ptz9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:40.917587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:40.917573 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018214 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptz9j\" (UniqueName: \"kubernetes.io/projected/b98fc85f-290e-42a0-88c9-70ec55212176-kube-api-access-ptz9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018273 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018305 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018302 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018645 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018325 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98fc85f-290e-42a0-88c9-70ec55212176-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018645 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018361 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018645 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018515 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018777 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018759 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018843 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018823 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.018909 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.018891 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.020663 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.020646 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b98fc85f-290e-42a0-88c9-70ec55212176-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.020770 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.020753 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b98fc85f-290e-42a0-88c9-70ec55212176-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.025665 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.025647 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptz9j\" (UniqueName: \"kubernetes.io/projected/b98fc85f-290e-42a0-88c9-70ec55212176-kube-api-access-ptz9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vp2n9\" (UID: \"b98fc85f-290e-42a0-88c9-70ec55212176\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.064789 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.064754 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:41.201554 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.201526 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9"] Jul 24 22:57:41.204249 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:57:41.204222 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98fc85f_290e_42a0_88c9_70ec55212176.slice/crio-e2a3df28a97bdfbbf5a7bc984845756e26444d7ce4a4b4a2977edf47fc4e386b WatchSource:0}: Error finding container e2a3df28a97bdfbbf5a7bc984845756e26444d7ce4a4b4a2977edf47fc4e386b: Status 404 returned error can't find the container with id e2a3df28a97bdfbbf5a7bc984845756e26444d7ce4a4b4a2977edf47fc4e386b Jul 24 22:57:41.205587 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.205567 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 22:57:41.945303 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:41.945267 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" event={"ID":"b98fc85f-290e-42a0-88c9-70ec55212176","Type":"ContainerStarted","Data":"e2a3df28a97bdfbbf5a7bc984845756e26444d7ce4a4b4a2977edf47fc4e386b"} Jul 24 22:57:42.952204 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:42.952170 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" event={"ID":"b98fc85f-290e-42a0-88c9-70ec55212176","Type":"ContainerStarted","Data":"9f630241ab14bae6f415959b896492d07a6c9b2cc6b7ef178d56dea21a93a58d"} Jul 24 22:57:42.967271 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:42.967221 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" podStartSLOduration=1.949475815 podStartE2EDuration="2.967205527s" totalImagesPullingTime="1.017729712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:57:42.9662564 +0000 UTC m=+691.895595611" watchObservedRunningTime="2026-07-24 22:57:42.967205527 +0000 UTC m=+691.896544739" Jul 24 22:57:51.065563 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:51.065517 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:51.065973 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:51.065624 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:51.077820 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:51.077792 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:52.009901 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:52.009870 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vp2n9" Jul 24 22:57:59.161416 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.161365 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7"] Jul 24 22:57:59.165129 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.165109 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.166910 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.166892 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 22:57:59.174797 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.174774 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7"] Jul 24 22:57:59.186415 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186381 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.186599 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186579 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95cjj\" (UniqueName: \"kubernetes.io/projected/70bfc930-2e9c-46ea-936d-26f47ccacd14-kube-api-access-95cjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.186687 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186632 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.186687 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186664 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.186782 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186720 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bfc930-2e9c-46ea-936d-26f47ccacd14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.186825 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.186801 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287416 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287354 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bfc930-2e9c-46ea-936d-26f47ccacd14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287426 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287460 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287554 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95cjj\" (UniqueName: \"kubernetes.io/projected/70bfc930-2e9c-46ea-936d-26f47ccacd14-kube-api-access-95cjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287601 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287590 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.287807 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287616 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.288012 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.287981 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.288148 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.288070 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.288202 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.288162 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.289813 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.289792 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70bfc930-2e9c-46ea-936d-26f47ccacd14-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.289961 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.289943 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70bfc930-2e9c-46ea-936d-26f47ccacd14-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.294764 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.294743 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95cjj\" (UniqueName: \"kubernetes.io/projected/70bfc930-2e9c-46ea-936d-26f47ccacd14-kube-api-access-95cjj\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7\" (UID: \"70bfc930-2e9c-46ea-936d-26f47ccacd14\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.476237 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.476150 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:57:59.609005 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:57:59.608979 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7"] Jul 24 22:57:59.611032 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:57:59.611007 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bfc930_2e9c_46ea_936d_26f47ccacd14.slice/crio-09fb359db1926c6d50299ab6413558ca58a95f7f95349c9ce1072f6cb2e2aba0 WatchSource:0}: Error finding container 09fb359db1926c6d50299ab6413558ca58a95f7f95349c9ce1072f6cb2e2aba0: Status 404 returned error can't find the container with id 09fb359db1926c6d50299ab6413558ca58a95f7f95349c9ce1072f6cb2e2aba0 Jul 24 22:58:00.039070 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:00.039027 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" event={"ID":"70bfc930-2e9c-46ea-936d-26f47ccacd14","Type":"ContainerStarted","Data":"028038291d8f3febd349d97f95445a0118045a12d4709fe60298047065e67fe6"} Jul 24 22:58:00.039241 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:00.039086 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" event={"ID":"70bfc930-2e9c-46ea-936d-26f47ccacd14","Type":"ContainerStarted","Data":"09fb359db1926c6d50299ab6413558ca58a95f7f95349c9ce1072f6cb2e2aba0"} Jul 24 22:58:00.054346 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:00.054294 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" podStartSLOduration=0.842008741 podStartE2EDuration="1.054276173s" totalImagesPullingTime="212.267432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:57:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:58:00.054044006 +0000 UTC m=+708.983383218" watchObservedRunningTime="2026-07-24 22:58:00.054276173 +0000 UTC m=+708.983615384" Jul 24 22:58:09.476569 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:09.476474 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:58:09.476569 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:09.476551 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:58:09.489610 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:09.489580 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:58:10.097494 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:10.097465 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7" Jul 24 22:58:30.540006 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.539966 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j"] Jul 24 22:58:30.543560 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.543541 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.545480 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.545458 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 24 22:58:30.553628 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.553603 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j"] Jul 24 22:58:30.588899 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.588865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd7d196-5a8b-45c6-8306-a72e216f1904-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.589062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.588907 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.589062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.588932 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5xnl\" (UniqueName: \"kubernetes.io/projected/cbd7d196-5a8b-45c6-8306-a72e216f1904-kube-api-access-n5xnl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.589062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.588953 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.589062 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.588994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.589198 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.589055 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690018 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.689976 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd7d196-5a8b-45c6-8306-a72e216f1904-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690026 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690060 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5xnl\" (UniqueName: \"kubernetes.io/projected/cbd7d196-5a8b-45c6-8306-a72e216f1904-kube-api-access-n5xnl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690089 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690140 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690191 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690184 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690559 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690520 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690682 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690574 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.690682 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.690672 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.692334 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.692313 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cbd7d196-5a8b-45c6-8306-a72e216f1904-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.692527 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.692510 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cbd7d196-5a8b-45c6-8306-a72e216f1904-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.697292 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.697267 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5xnl\" (UniqueName: \"kubernetes.io/projected/cbd7d196-5a8b-45c6-8306-a72e216f1904-kube-api-access-n5xnl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cw25j\" (UID: \"cbd7d196-5a8b-45c6-8306-a72e216f1904\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.853193 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.853103 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:30.979497 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:30.979473 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j"] Jul 24 22:58:30.981695 ip-10-0-136-126 kubenswrapper[2571]: W0724 22:58:30.981656 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcbd7d196_5a8b_45c6_8306_a72e216f1904.slice/crio-489e669fd8d615b5d0e644ffc154a127538bdc579963ead5811fcc361a3e7868 WatchSource:0}: Error finding container 489e669fd8d615b5d0e644ffc154a127538bdc579963ead5811fcc361a3e7868: Status 404 returned error can't find the container with id 489e669fd8d615b5d0e644ffc154a127538bdc579963ead5811fcc361a3e7868 Jul 24 22:58:31.187918 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:31.187872 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" event={"ID":"cbd7d196-5a8b-45c6-8306-a72e216f1904","Type":"ContainerStarted","Data":"489e669fd8d615b5d0e644ffc154a127538bdc579963ead5811fcc361a3e7868"} Jul 24 22:58:32.193723 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:32.193686 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" event={"ID":"cbd7d196-5a8b-45c6-8306-a72e216f1904","Type":"ContainerStarted","Data":"7f2b5939720a7879f82e87daea839a58d9dff0ff1de3cc79b9a32495cc8705a3"} Jul 24 22:58:32.210644 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:32.210595 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" podStartSLOduration=2.043131233 podStartE2EDuration="2.21058035s" totalImagesPullingTime="167.449117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 22:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 22:58:32.209409108 +0000 UTC m=+741.138748319" watchObservedRunningTime="2026-07-24 22:58:32.21058035 +0000 UTC m=+741.139919560" Jul 24 22:58:40.853590 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:40.853555 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:40.853979 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:40.853620 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:40.866183 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:40.866156 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 22:58:41.245010 ip-10-0-136-126 kubenswrapper[2571]: I0724 22:58:41.244934 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cw25j" Jul 24 23:00:24.322977 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.322942 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 23:00:24.323498 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.323160 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9dbb4674-nfs8w" podUID="560d79f6-7643-46e1-9456-44da9e8ec355" containerName="manager" containerID="cri-o://b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a" gracePeriod=30 Jul 24 23:00:24.575864 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.575808 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 23:00:24.683497 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.683465 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7wgf\" (UniqueName: \"kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf\") pod \"560d79f6-7643-46e1-9456-44da9e8ec355\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " Jul 24 23:00:24.683666 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.683582 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert\") pod \"560d79f6-7643-46e1-9456-44da9e8ec355\" (UID: \"560d79f6-7643-46e1-9456-44da9e8ec355\") " Jul 24 23:00:24.685639 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.685606 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "560d79f6-7643-46e1-9456-44da9e8ec355" (UID: "560d79f6-7643-46e1-9456-44da9e8ec355"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:00:24.685639 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.685611 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf" (OuterVolumeSpecName: "kube-api-access-j7wgf") pod "560d79f6-7643-46e1-9456-44da9e8ec355" (UID: "560d79f6-7643-46e1-9456-44da9e8ec355"). InnerVolumeSpecName "kube-api-access-j7wgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:00:24.720405 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.720372 2571 generic.go:356] "Generic (PLEG): container finished" podID="560d79f6-7643-46e1-9456-44da9e8ec355" containerID="b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a" exitCode=0 Jul 24 23:00:24.720534 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.720434 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-nfs8w" Jul 24 23:00:24.720534 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.720464 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-nfs8w" event={"ID":"560d79f6-7643-46e1-9456-44da9e8ec355","Type":"ContainerDied","Data":"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a"} Jul 24 23:00:24.720534 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.720530 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-nfs8w" event={"ID":"560d79f6-7643-46e1-9456-44da9e8ec355","Type":"ContainerDied","Data":"d63fa4a228071093f655479ed56b3e27629d5d8dc569177242cd5d0852f1c8d6"} Jul 24 23:00:24.720694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.720563 2571 scope.go:117] "RemoveContainer" containerID="b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a" Jul 24 23:00:24.733137 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.733119 2571 scope.go:117] "RemoveContainer" containerID="b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a" Jul 24 23:00:24.733441 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:00:24.733418 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a\": container with ID starting with b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a not found: ID does not exist" containerID="b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a" Jul 24 23:00:24.733526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.733450 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a"} err="failed to get container status \"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a\": rpc error: code = NotFound desc = could not find container \"b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a\": container with ID starting with b9d52be15ab626a878b35b667060d07c9bbb68a6361e0a3ca7686bb8a721ab1a not found: ID does not exist" Jul 24 23:00:24.744570 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.744544 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 23:00:24.747857 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.747837 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9dbb4674-nfs8w"] Jul 24 23:00:24.784353 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.784325 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/560d79f6-7643-46e1-9456-44da9e8ec355-webhook-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:00:24.784353 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:24.784353 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7wgf\" (UniqueName: \"kubernetes.io/projected/560d79f6-7643-46e1-9456-44da9e8ec355-kube-api-access-j7wgf\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:00:25.559981 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:00:25.559945 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="560d79f6-7643-46e1-9456-44da9e8ec355" path="/var/lib/kubelet/pods/560d79f6-7643-46e1-9456-44da9e8ec355/volumes" Jul 24 23:01:12.345974 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:12.345948 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:01:12.347140 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:12.347120 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:01:12.351186 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:12.351163 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:01:12.352157 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:12.352136 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:01:25.544202 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.544159 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:25.544686 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.544584 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="560d79f6-7643-46e1-9456-44da9e8ec355" containerName="manager" Jul 24 23:01:25.544686 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.544596 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="560d79f6-7643-46e1-9456-44da9e8ec355" containerName="manager" Jul 24 23:01:25.544686 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.544676 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="560d79f6-7643-46e1-9456-44da9e8ec355" containerName="manager" Jul 24 23:01:25.548870 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.548851 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.551184 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.551160 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kgsz\"" Jul 24 23:01:25.551308 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.551228 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 23:01:25.551308 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.551254 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 23:01:25.562667 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.562640 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:25.655314 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.655283 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.655471 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.655372 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ggv\" (UniqueName: \"kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.756619 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.756586 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ggv\" (UniqueName: \"kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.756769 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.756738 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.759104 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.759076 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.763648 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.763626 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ggv\" (UniqueName: \"kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv\") pod \"maas-controller-9dbb4674-57tzr\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:25.861190 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:25.861111 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:26.198352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:26.198272 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:26.202007 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:01:26.201968 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff0f4be_a029_4a67_89d7_dc13de988a2d.slice/crio-9d2f908539a5703c6b7ebaf6e06917901805ced422b859abc4c112d64140f6f0 WatchSource:0}: Error finding container 9d2f908539a5703c6b7ebaf6e06917901805ced422b859abc4c112d64140f6f0: Status 404 returned error can't find the container with id 9d2f908539a5703c6b7ebaf6e06917901805ced422b859abc4c112d64140f6f0 Jul 24 23:01:27.026494 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:27.026455 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-57tzr" event={"ID":"7ff0f4be-a029-4a67-89d7-dc13de988a2d","Type":"ContainerStarted","Data":"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a"} Jul 24 23:01:27.026984 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:27.026507 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-57tzr" event={"ID":"7ff0f4be-a029-4a67-89d7-dc13de988a2d","Type":"ContainerStarted","Data":"9d2f908539a5703c6b7ebaf6e06917901805ced422b859abc4c112d64140f6f0"} Jul 24 23:01:27.026984 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:27.026561 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:27.042120 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:27.042063 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9dbb4674-57tzr" podStartSLOduration=1.42639038 podStartE2EDuration="2.042039938s" totalImagesPullingTime="615.649558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:01:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:01:27.040369507 +0000 UTC m=+915.969708718" watchObservedRunningTime="2026-07-24 23:01:27.042039938 +0000 UTC m=+915.971379154" Jul 24 23:01:38.033292 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:38.033254 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:42.669407 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:42.669361 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:42.669834 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:42.669558 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-9dbb4674-57tzr" podUID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" containerName="manager" containerID="cri-o://95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a" gracePeriod=30 Jul 24 23:01:42.920473 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:42.920414 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:43.020767 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.020728 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6ggv\" (UniqueName: \"kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv\") pod \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " Jul 24 23:01:43.020970 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.020811 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert\") pod \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\" (UID: \"7ff0f4be-a029-4a67-89d7-dc13de988a2d\") " Jul 24 23:01:43.022831 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.022801 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7ff0f4be-a029-4a67-89d7-dc13de988a2d" (UID: "7ff0f4be-a029-4a67-89d7-dc13de988a2d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:01:43.022831 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.022824 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv" (OuterVolumeSpecName: "kube-api-access-f6ggv") pod "7ff0f4be-a029-4a67-89d7-dc13de988a2d" (UID: "7ff0f4be-a029-4a67-89d7-dc13de988a2d"). InnerVolumeSpecName "kube-api-access-f6ggv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:01:43.106382 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.106340 2571 generic.go:356] "Generic (PLEG): container finished" podID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" containerID="95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a" exitCode=0 Jul 24 23:01:43.106607 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.106427 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-57tzr" Jul 24 23:01:43.106607 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.106433 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-57tzr" event={"ID":"7ff0f4be-a029-4a67-89d7-dc13de988a2d","Type":"ContainerDied","Data":"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a"} Jul 24 23:01:43.106607 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.106493 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-57tzr" event={"ID":"7ff0f4be-a029-4a67-89d7-dc13de988a2d","Type":"ContainerDied","Data":"9d2f908539a5703c6b7ebaf6e06917901805ced422b859abc4c112d64140f6f0"} Jul 24 23:01:43.106607 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.106516 2571 scope.go:117] "RemoveContainer" containerID="95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a" Jul 24 23:01:43.116314 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.116291 2571 scope.go:117] "RemoveContainer" containerID="95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a" Jul 24 23:01:43.116613 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:01:43.116594 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a\": container with ID starting with 95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a not found: ID does not exist" containerID="95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a" Jul 24 23:01:43.116668 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.116622 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a"} err="failed to get container status \"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a\": rpc error: code = NotFound desc = could not find container \"95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a\": container with ID starting with 95ddc6751e3411b2f0cf81f1213e3325b143c7e86e03370203f1a918871ac83a not found: ID does not exist" Jul 24 23:01:43.122313 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.122291 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ggv\" (UniqueName: \"kubernetes.io/projected/7ff0f4be-a029-4a67-89d7-dc13de988a2d-kube-api-access-f6ggv\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:01:43.122429 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.122318 2571 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ff0f4be-a029-4a67-89d7-dc13de988a2d-webhook-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:01:43.127112 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.127087 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:43.133029 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.133008 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-9dbb4674-57tzr"] Jul 24 23:01:43.566441 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.566385 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" path="/var/lib/kubelet/pods/7ff0f4be-a029-4a67-89d7-dc13de988a2d/volumes" Jul 24 23:01:43.922072 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.921986 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-9dbb4674-s44gl"] Jul 24 23:01:43.922448 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.922402 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" containerName="manager" Jul 24 23:01:43.922448 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.922414 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" containerName="manager" Jul 24 23:01:43.922530 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.922480 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff0f4be-a029-4a67-89d7-dc13de988a2d" containerName="manager" Jul 24 23:01:43.926764 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.926745 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:43.928900 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.928880 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6kgsz\"" Jul 24 23:01:43.929234 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.929214 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 24 23:01:43.929352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.929247 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 24 23:01:43.936285 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:43.936259 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-s44gl"] Jul 24 23:01:44.030992 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.030947 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-webhook-cert\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.031185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.031027 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg4t6\" (UniqueName: \"kubernetes.io/projected/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-kube-api-access-xg4t6\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.131734 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.131672 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-webhook-cert\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.131949 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.131757 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg4t6\" (UniqueName: \"kubernetes.io/projected/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-kube-api-access-xg4t6\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.134070 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.134049 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-webhook-cert\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.138576 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.138557 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg4t6\" (UniqueName: \"kubernetes.io/projected/524fb69a-b2e2-4838-8e37-c4fbe1270a5f-kube-api-access-xg4t6\") pod \"maas-controller-9dbb4674-s44gl\" (UID: \"524fb69a-b2e2-4838-8e37-c4fbe1270a5f\") " pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.237644 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.237546 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:44.372097 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:44.372058 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-9dbb4674-s44gl"] Jul 24 23:01:44.375192 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:01:44.375162 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524fb69a_b2e2_4838_8e37_c4fbe1270a5f.slice/crio-f2cc9b1a8d28df5dd7c9a82b7240a26924af1bb1ae02c625524e0ad8cbdf64ee WatchSource:0}: Error finding container f2cc9b1a8d28df5dd7c9a82b7240a26924af1bb1ae02c625524e0ad8cbdf64ee: Status 404 returned error can't find the container with id f2cc9b1a8d28df5dd7c9a82b7240a26924af1bb1ae02c625524e0ad8cbdf64ee Jul 24 23:01:45.118735 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:45.118693 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-s44gl" event={"ID":"524fb69a-b2e2-4838-8e37-c4fbe1270a5f","Type":"ContainerStarted","Data":"ec852fb5479e6d8b48de263856ab1b49b7119038cdcfb4682b2598a68201ba5d"} Jul 24 23:01:45.119159 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:45.118753 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-9dbb4674-s44gl" event={"ID":"524fb69a-b2e2-4838-8e37-c4fbe1270a5f","Type":"ContainerStarted","Data":"f2cc9b1a8d28df5dd7c9a82b7240a26924af1bb1ae02c625524e0ad8cbdf64ee"} Jul 24 23:01:45.119159 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:45.118798 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:01:45.135663 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:45.135598 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-9dbb4674-s44gl" podStartSLOduration=1.784263717 podStartE2EDuration="2.135581547s" totalImagesPullingTime="351.31783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:01:45.135194795 +0000 UTC m=+934.064534007" watchObservedRunningTime="2026-07-24 23:01:45.135581547 +0000 UTC m=+934.064920761" Jul 24 23:01:56.125501 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:01:56.125467 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-9dbb4674-s44gl" Jul 24 23:06:12.380670 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:06:12.380636 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:06:12.384253 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:06:12.384227 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:06:12.386536 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:06:12.386513 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:06:12.389695 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:06:12.389674 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:10:36.266731 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.266694 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:10:36.270229 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.270205 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.272383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.272358 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e1e63fa5-kserve-self-signed-certs\"" Jul 24 23:10:36.280612 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.280591 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:10:36.306290 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306256 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.306466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306294 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.306466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.306466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306430 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.306634 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306485 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf2cv\" (UniqueName: \"kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.306634 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.306542 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407408 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407369 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407600 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407468 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf2cv\" (UniqueName: \"kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407600 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407491 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407713 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407686 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407771 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407726 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407771 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407748 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.407876 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407775 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.408015 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.407998 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.408056 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.408035 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.409741 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.409716 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.410386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.410364 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.415944 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.415920 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf2cv\" (UniqueName: \"kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv\") pod \"e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.580788 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.580751 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:36.718706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:36.718680 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:10:36.721472 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:10:36.721448 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bafd412_95d2_4764_8344_ca2036372e10.slice/crio-d37a488b9691d4254a573d161889f4086e17fd0b5348b68aea6d7e4cca7c51d3 WatchSource:0}: Error finding container d37a488b9691d4254a573d161889f4086e17fd0b5348b68aea6d7e4cca7c51d3: Status 404 returned error can't find the container with id d37a488b9691d4254a573d161889f4086e17fd0b5348b68aea6d7e4cca7c51d3 Jul 24 23:10:37.706142 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:37.706107 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" event={"ID":"9bafd412-95d2-4764-8344-ca2036372e10","Type":"ContainerStarted","Data":"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f"} Jul 24 23:10:37.706549 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:37.706153 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" event={"ID":"9bafd412-95d2-4764-8344-ca2036372e10","Type":"ContainerStarted","Data":"d37a488b9691d4254a573d161889f4086e17fd0b5348b68aea6d7e4cca7c51d3"} Jul 24 23:10:37.721043 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:37.720993 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" podStartSLOduration=1.7209778359999999 podStartE2EDuration="1.720977836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:10:37.719618831 +0000 UTC m=+1466.648958033" watchObservedRunningTime="2026-07-24 23:10:37.720977836 +0000 UTC m=+1466.650317046" Jul 24 23:10:46.581546 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:46.581500 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:46.581971 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:46.581566 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:46.594317 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:46.594280 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:10:46.767014 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:10:46.766980 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:11:08.540353 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:08.540302 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:11:12.421387 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:12.421357 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:11:12.426945 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:12.426920 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:11:12.428115 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:12.428094 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:11:12.433294 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:12.433276 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:11:23.559903 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.559845 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" podUID="9bafd412-95d2-4764-8344-ca2036372e10" containerName="main" containerID="cri-o://c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f" gracePeriod=45 Jul 24 23:11:23.804206 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.804173 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:11:23.882018 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.881930 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882045 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882090 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882126 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882168 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882346 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882205 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf2cv\" (UniqueName: \"kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv\") pod \"9bafd412-95d2-4764-8344-ca2036372e10\" (UID: \"9bafd412-95d2-4764-8344-ca2036372e10\") " Jul 24 23:11:23.882426 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882366 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home" (OuterVolumeSpecName: "home") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:11:23.882426 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882365 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:11:23.882539 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882388 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache" (OuterVolumeSpecName: "model-cache") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:11:23.882636 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882612 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:23.882636 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882636 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:23.882878 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.882652 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:23.884269 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.884247 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm" (OuterVolumeSpecName: "dshm") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:11:23.884350 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.884247 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv" (OuterVolumeSpecName: "kube-api-access-pf2cv") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "kube-api-access-pf2cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:11:23.884350 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.884317 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9bafd412-95d2-4764-8344-ca2036372e10" (UID: "9bafd412-95d2-4764-8344-ca2036372e10"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:11:23.942388 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.942348 2571 generic.go:356] "Generic (PLEG): container finished" podID="9bafd412-95d2-4764-8344-ca2036372e10" containerID="c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f" exitCode=0 Jul 24 23:11:23.942604 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.942520 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" event={"ID":"9bafd412-95d2-4764-8344-ca2036372e10","Type":"ContainerDied","Data":"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f"} Jul 24 23:11:23.942604 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.942570 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" event={"ID":"9bafd412-95d2-4764-8344-ca2036372e10","Type":"ContainerDied","Data":"d37a488b9691d4254a573d161889f4086e17fd0b5348b68aea6d7e4cca7c51d3"} Jul 24 23:11:23.942604 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.942603 2571 scope.go:117] "RemoveContainer" containerID="c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f" Jul 24 23:11:23.943025 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.942816 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl" Jul 24 23:11:23.953730 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.953711 2571 scope.go:117] "RemoveContainer" containerID="c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f" Jul 24 23:11:23.954006 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:11:23.953981 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f\": container with ID starting with c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f not found: ID does not exist" containerID="c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f" Jul 24 23:11:23.954070 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.954018 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f"} err="failed to get container status \"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f\": rpc error: code = NotFound desc = could not find container \"c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f\": container with ID starting with c0a4f7fdf7e80d1c0593bea9224b4bc54b362bd19a9db14045b71847d4d8eb8f not found: ID does not exist" Jul 24 23:11:23.964315 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.964292 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:11:23.968813 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.968792 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e1e63fa5-kserve-76ff954999-9xdzl"] Jul 24 23:11:23.983810 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.983793 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9bafd412-95d2-4764-8344-ca2036372e10-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:23.983874 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.983813 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf2cv\" (UniqueName: \"kubernetes.io/projected/9bafd412-95d2-4764-8344-ca2036372e10-kube-api-access-pf2cv\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:23.983874 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:23.983822 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9bafd412-95d2-4764-8344-ca2036372e10-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:11:25.560776 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:11:25.560742 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bafd412-95d2-4764-8344-ca2036372e10" path="/var/lib/kubelet/pods/9bafd412-95d2-4764-8344-ca2036372e10/volumes" Jul 24 23:16:12.460226 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:16:12.460194 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:16:12.463982 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:16:12.463961 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:16:12.465349 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:16:12.465321 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:16:12.468912 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:16:12.468895 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:21:12.491804 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:21:12.491768 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:21:12.496583 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:21:12.496557 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:21:12.497041 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:21:12.497006 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:21:12.501760 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:21:12.501731 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:23:38.750912 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.750832 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:23:38.751362 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.751292 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bafd412-95d2-4764-8344-ca2036372e10" containerName="main" Jul 24 23:23:38.751362 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.751306 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bafd412-95d2-4764-8344-ca2036372e10" containerName="main" Jul 24 23:23:38.751470 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.751374 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bafd412-95d2-4764-8344-ca2036372e10" containerName="main" Jul 24 23:23:38.754833 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.754812 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.757061 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.757038 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b391a\"/\"test-model-3b391a-kserve-self-signed-certs\"" Jul 24 23:23:38.757191 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.757085 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3b391a\"/\"default-dockercfg-rpljt\"" Jul 24 23:23:38.757191 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.757099 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b391a\"/\"kube-root-ca.crt\"" Jul 24 23:23:38.757191 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.757100 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3b391a\"/\"openshift-service-ca.crt\"" Jul 24 23:23:38.762316 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.762015 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:23:38.879384 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879349 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.879384 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879382 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px75m\" (UniqueName: \"kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.879675 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879500 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.879675 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.879675 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879609 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.879823 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.879718 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.980838 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980806 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980856 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980873 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px75m\" (UniqueName: \"kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980899 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980922 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.980948 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981304 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.981254 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981304 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.981240 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.981408 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.981309 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.983027 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.982992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.983231 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.983214 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:38.988029 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:38.987997 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px75m\" (UniqueName: \"kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m\") pod \"test-model-3b391a-kserve-67cddcdfd7-xx78d\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:39.065089 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:39.065051 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:39.196419 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:39.196380 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:23:39.198240 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:23:39.198201 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b075d49_4a9f_4473_a069_da97e4e65454.slice/crio-f47c34ad824ae7da8998650084da36899a0ccbf9ab1b057fad4a01e8a769e570 WatchSource:0}: Error finding container f47c34ad824ae7da8998650084da36899a0ccbf9ab1b057fad4a01e8a769e570: Status 404 returned error can't find the container with id f47c34ad824ae7da8998650084da36899a0ccbf9ab1b057fad4a01e8a769e570 Jul 24 23:23:39.560103 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:39.560067 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" event={"ID":"6b075d49-4a9f-4473-a069-da97e4e65454","Type":"ContainerStarted","Data":"3b9ff0fbd18bb312f7df37017d7e1067493b4cc6de48de0047117bc90334a2f0"} Jul 24 23:23:39.560103 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:39.560105 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" event={"ID":"6b075d49-4a9f-4473-a069-da97e4e65454","Type":"ContainerStarted","Data":"f47c34ad824ae7da8998650084da36899a0ccbf9ab1b057fad4a01e8a769e570"} Jul 24 23:23:39.575205 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:39.575154 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" podStartSLOduration=1.575139022 podStartE2EDuration="1.575139022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:23:39.573683763 +0000 UTC m=+2248.503022973" watchObservedRunningTime="2026-07-24 23:23:39.575139022 +0000 UTC m=+2248.504478234" Jul 24 23:23:49.065687 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:49.065654 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:49.066087 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:49.065703 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:49.078424 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:49.078378 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:23:49.619102 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:23:49.619072 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:24:04.062087 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.062046 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:24:04.066274 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.066251 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.068175 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.068156 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-313c6a\"/\"kube-root-ca.crt\"" Jul 24 23:24:04.068708 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.068688 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-313c6a\"/\"openshift-service-ca.crt\"" Jul 24 23:24:04.068779 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.068726 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-313c6a\"/\"test-model-313c6a-kserve-self-signed-certs\"" Jul 24 23:24:04.068779 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.068734 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-313c6a\"/\"default-dockercfg-rgdvn\"" Jul 24 23:24:04.074986 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.074962 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:24:04.232847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.232809 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg9mg\" (UniqueName: \"kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.232847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.232849 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.233059 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.232950 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.233059 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.232991 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.233059 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.233017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.233156 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.233129 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.333906 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.333822 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.333906 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.333868 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg9mg\" (UniqueName: \"kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.333906 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.333888 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334167 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.333953 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334167 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.333983 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334167 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.334010 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334361 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.334334 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334458 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.334348 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.334458 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.334437 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.336184 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.336163 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.336507 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.336489 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.341084 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.341060 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg9mg\" (UniqueName: \"kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg\") pod \"test-model-313c6a-kserve-7d76f47d55-njlw9\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.377086 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.377053 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:04.513886 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.513861 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:24:04.516377 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:24:04.516346 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80e867a7_de3d_455f_aa42_c249dc3d4dd0.slice/crio-409a35f6377bc9f96c169be59bf93b4a493dd5e0500ca72e5a5449d6db61d513 WatchSource:0}: Error finding container 409a35f6377bc9f96c169be59bf93b4a493dd5e0500ca72e5a5449d6db61d513: Status 404 returned error can't find the container with id 409a35f6377bc9f96c169be59bf93b4a493dd5e0500ca72e5a5449d6db61d513 Jul 24 23:24:04.682186 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.682104 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" event={"ID":"80e867a7-de3d-455f-aa42-c249dc3d4dd0","Type":"ContainerStarted","Data":"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b"} Jul 24 23:24:04.682186 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.682146 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" event={"ID":"80e867a7-de3d-455f-aa42-c249dc3d4dd0","Type":"ContainerStarted","Data":"409a35f6377bc9f96c169be59bf93b4a493dd5e0500ca72e5a5449d6db61d513"} Jul 24 23:24:04.697525 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:04.697478 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" podStartSLOduration=0.697463052 podStartE2EDuration="697.463052ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:24:04.696632996 +0000 UTC m=+2273.625972207" watchObservedRunningTime="2026-07-24 23:24:04.697463052 +0000 UTC m=+2273.626802295" Jul 24 23:24:14.378106 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:14.378069 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:14.378697 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:14.378122 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:14.390376 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:14.390352 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:24:14.744487 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:24:14.744384 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:25:04.882113 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:04.882024 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:25:05.766834 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:05.766800 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:25:05.916822 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:05.916790 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp"] Jul 24 23:25:05.920661 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:05.920637 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:05.922362 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:05.922338 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\"" Jul 24 23:25:05.925298 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:05.925267 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp"] Jul 24 23:25:06.033330 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.033300 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp\" (UID: \"ffa03203-36dc-499b-a3a3-e6157ff5ede2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:06.133958 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.133917 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp\" (UID: \"ffa03203-36dc-499b-a3a3-e6157ff5ede2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:06.134619 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.134600 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp\" (UID: \"ffa03203-36dc-499b-a3a3-e6157ff5ede2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:06.234589 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.234556 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:06.369952 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.369921 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp"] Jul 24 23:25:06.373852 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:25:06.373822 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa03203_36dc_499b_a3a3_e6157ff5ede2.slice/crio-42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b WatchSource:0}: Error finding container 42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b: Status 404 returned error can't find the container with id 42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b Jul 24 23:25:06.376303 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:06.376282 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:25:07.001843 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:07.001803 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" event={"ID":"ffa03203-36dc-499b-a3a3-e6157ff5ede2","Type":"ContainerStarted","Data":"42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b"} Jul 24 23:25:09.014471 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:09.014428 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" event={"ID":"ffa03203-36dc-499b-a3a3-e6157ff5ede2","Type":"ContainerStarted","Data":"0f1f1ea8f7f8e78357e64a560969ca0c0cb6b39bc36fe88eafc48157c7473e4a"} Jul 24 23:25:09.027179 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:09.027120 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" podStartSLOduration=1.7138005170000001 podStartE2EDuration="4.027101408s" totalImagesPullingTime="2.313300891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:25:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:25:09.026280979 +0000 UTC m=+2337.955620191" watchObservedRunningTime="2026-07-24 23:25:09.027101408 +0000 UTC m=+2337.956440624" Jul 24 23:25:10.022701 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:10.022630 2571 generic.go:356] "Generic (PLEG): container finished" podID="ffa03203-36dc-499b-a3a3-e6157ff5ede2" containerID="0f1f1ea8f7f8e78357e64a560969ca0c0cb6b39bc36fe88eafc48157c7473e4a" exitCode=0 Jul 24 23:25:10.023127 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:10.022722 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" event={"ID":"ffa03203-36dc-499b-a3a3-e6157ff5ede2","Type":"ContainerDied","Data":"0f1f1ea8f7f8e78357e64a560969ca0c0cb6b39bc36fe88eafc48157c7473e4a"} Jul 24 23:25:11.162991 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:11.162966 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:11.282239 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:11.282148 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca\") pod \"ffa03203-36dc-499b-a3a3-e6157ff5ede2\" (UID: \"ffa03203-36dc-499b-a3a3-e6157ff5ede2\") " Jul 24 23:25:11.282553 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:11.282528 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ffa03203-36dc-499b-a3a3-e6157ff5ede2" (UID: "ffa03203-36dc-499b-a3a3-e6157ff5ede2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:25:11.383248 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:11.383212 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffa03203-36dc-499b-a3a3-e6157ff5ede2-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:12.035550 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:12.035515 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" Jul 24 23:25:12.035731 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:12.035552 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp" event={"ID":"ffa03203-36dc-499b-a3a3-e6157ff5ede2","Type":"ContainerDied","Data":"42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b"} Jul 24 23:25:12.035731 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:12.035604 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f1ab4ca8545abbb67916f044ac89d39b03bf211102927b354d8c6feb84bd8b" Jul 24 23:25:19.903301 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:19.903251 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" podUID="6b075d49-4a9f-4473-a069-da97e4e65454" containerName="main" containerID="cri-o://3b9ff0fbd18bb312f7df37017d7e1067493b4cc6de48de0047117bc90334a2f0" gracePeriod=45 Jul 24 23:25:20.076879 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.076844 2571 generic.go:356] "Generic (PLEG): container finished" podID="6b075d49-4a9f-4473-a069-da97e4e65454" containerID="3b9ff0fbd18bb312f7df37017d7e1067493b4cc6de48de0047117bc90334a2f0" exitCode=0 Jul 24 23:25:20.077008 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.076894 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" event={"ID":"6b075d49-4a9f-4473-a069-da97e4e65454","Type":"ContainerDied","Data":"3b9ff0fbd18bb312f7df37017d7e1067493b4cc6de48de0047117bc90334a2f0"} Jul 24 23:25:20.156345 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.156275 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:25:20.270843 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.270797 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271050 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.270886 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271050 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.270997 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px75m\" (UniqueName: \"kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271050 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271032 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271209 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271052 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271209 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271063 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home" (OuterVolumeSpecName: "home") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:20.271209 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271086 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs\") pod \"6b075d49-4a9f-4473-a069-da97e4e65454\" (UID: \"6b075d49-4a9f-4473-a069-da97e4e65454\") " Jul 24 23:25:20.271209 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271160 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:20.271383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271329 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:20.271521 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271495 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.271521 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271521 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.271643 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.271530 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.273266 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.273246 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm" (OuterVolumeSpecName: "dshm") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:20.273384 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.273289 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m" (OuterVolumeSpecName: "kube-api-access-px75m") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "kube-api-access-px75m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:25:20.273384 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.273365 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b075d49-4a9f-4473-a069-da97e4e65454" (UID: "6b075d49-4a9f-4473-a069-da97e4e65454"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:25:20.372586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.372541 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px75m\" (UniqueName: \"kubernetes.io/projected/6b075d49-4a9f-4473-a069-da97e4e65454-kube-api-access-px75m\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.372586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.372578 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b075d49-4a9f-4473-a069-da97e4e65454-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.372586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.372587 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b075d49-4a9f-4473-a069-da97e4e65454-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:20.786454 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.786417 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" podUID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" containerName="main" containerID="cri-o://65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b" gracePeriod=45 Jul 24 23:25:20.993691 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.993651 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp"] Jul 24 23:25:20.998028 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:20.998001 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3b391a-rcflp"] Jul 24 23:25:21.033632 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.033607 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:25:21.084185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.084106 2571 generic.go:356] "Generic (PLEG): container finished" podID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" containerID="65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b" exitCode=0 Jul 24 23:25:21.084185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.084167 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" Jul 24 23:25:21.084388 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.084199 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" event={"ID":"80e867a7-de3d-455f-aa42-c249dc3d4dd0","Type":"ContainerDied","Data":"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b"} Jul 24 23:25:21.084388 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.084247 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9" event={"ID":"80e867a7-de3d-455f-aa42-c249dc3d4dd0","Type":"ContainerDied","Data":"409a35f6377bc9f96c169be59bf93b4a493dd5e0500ca72e5a5449d6db61d513"} Jul 24 23:25:21.084388 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.084272 2571 scope.go:117] "RemoveContainer" containerID="65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b" Jul 24 23:25:21.085667 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.085632 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" event={"ID":"6b075d49-4a9f-4473-a069-da97e4e65454","Type":"ContainerDied","Data":"f47c34ad824ae7da8998650084da36899a0ccbf9ab1b057fad4a01e8a769e570"} Jul 24 23:25:21.085795 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.085670 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d" Jul 24 23:25:21.093386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.093371 2571 scope.go:117] "RemoveContainer" containerID="65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b" Jul 24 23:25:21.093695 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:25:21.093670 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b\": container with ID starting with 65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b not found: ID does not exist" containerID="65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b" Jul 24 23:25:21.093807 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.093705 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b"} err="failed to get container status \"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b\": rpc error: code = NotFound desc = could not find container \"65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b\": container with ID starting with 65f772d969c659147e1a55bbe010d83315adb092d361c049c833107cf8db1e9b not found: ID does not exist" Jul 24 23:25:21.093807 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.093730 2571 scope.go:117] "RemoveContainer" containerID="3b9ff0fbd18bb312f7df37017d7e1067493b4cc6de48de0047117bc90334a2f0" Jul 24 23:25:21.105349 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.105325 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:25:21.107098 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.107080 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3b391a/test-model-3b391a-kserve-67cddcdfd7-xx78d"] Jul 24 23:25:21.180748 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180709 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.180936 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180766 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.180936 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180823 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.180936 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180885 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.180936 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180915 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.181155 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180945 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg9mg\" (UniqueName: \"kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg\") pod \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\" (UID: \"80e867a7-de3d-455f-aa42-c249dc3d4dd0\") " Jul 24 23:25:21.181155 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.180999 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:21.181155 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.181080 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache" (OuterVolumeSpecName: "model-cache") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:21.181155 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.181095 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home" (OuterVolumeSpecName: "home") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:21.181345 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.181329 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.181412 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.181351 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.181412 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.181364 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.183166 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.183145 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm" (OuterVolumeSpecName: "dshm") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:25:21.183166 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.183148 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:25:21.183319 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.183253 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg" (OuterVolumeSpecName: "kube-api-access-sg9mg") pod "80e867a7-de3d-455f-aa42-c249dc3d4dd0" (UID: "80e867a7-de3d-455f-aa42-c249dc3d4dd0"). InnerVolumeSpecName "kube-api-access-sg9mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:25:21.282026 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.281986 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/80e867a7-de3d-455f-aa42-c249dc3d4dd0-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.282026 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.282019 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80e867a7-de3d-455f-aa42-c249dc3d4dd0-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.282026 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.282029 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg9mg\" (UniqueName: \"kubernetes.io/projected/80e867a7-de3d-455f-aa42-c249dc3d4dd0-kube-api-access-sg9mg\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:25:21.406408 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.406359 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:25:21.410574 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.410544 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-313c6a/test-model-313c6a-kserve-7d76f47d55-njlw9"] Jul 24 23:25:21.560591 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.560554 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b075d49-4a9f-4473-a069-da97e4e65454" path="/var/lib/kubelet/pods/6b075d49-4a9f-4473-a069-da97e4e65454/volumes" Jul 24 23:25:21.560968 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.560955 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" path="/var/lib/kubelet/pods/80e867a7-de3d-455f-aa42-c249dc3d4dd0/volumes" Jul 24 23:25:21.561249 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:25:21.561238 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa03203-36dc-499b-a3a3-e6157ff5ede2" path="/var/lib/kubelet/pods/ffa03203-36dc-499b-a3a3-e6157ff5ede2/volumes" Jul 24 23:26:12.532653 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:12.532614 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:26:12.538622 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:12.538587 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:26:12.538819 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:12.538800 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:26:12.544178 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:12.544160 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:26:13.899024 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.898987 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d"] Jul 24 23:26:13.899420 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899378 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" containerName="main" Jul 24 23:26:13.899420 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899406 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" containerName="main" Jul 24 23:26:13.899420 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899415 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa03203-36dc-499b-a3a3-e6157ff5ede2" containerName="revoke-keys" Jul 24 23:26:13.899420 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899421 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa03203-36dc-499b-a3a3-e6157ff5ede2" containerName="revoke-keys" Jul 24 23:26:13.899566 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899445 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b075d49-4a9f-4473-a069-da97e4e65454" containerName="main" Jul 24 23:26:13.899566 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899450 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b075d49-4a9f-4473-a069-da97e4e65454" containerName="main" Jul 24 23:26:13.899566 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899513 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b075d49-4a9f-4473-a069-da97e4e65454" containerName="main" Jul 24 23:26:13.899566 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899520 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="80e867a7-de3d-455f-aa42-c249dc3d4dd0" containerName="main" Jul 24 23:26:13.899566 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.899529 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa03203-36dc-499b-a3a3-e6157ff5ede2" containerName="revoke-keys" Jul 24 23:26:13.904127 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.904108 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:13.905957 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.905940 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\"" Jul 24 23:26:13.909465 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:13.909446 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d"] Jul 24 23:26:14.013368 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:14.013331 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d\" (UID: \"2c154516-0fb2-407e-bdc9-ccf98d5932e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:14.114947 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:14.114906 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d\" (UID: \"2c154516-0fb2-407e-bdc9-ccf98d5932e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:14.115577 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:14.115555 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d\" (UID: \"2c154516-0fb2-407e-bdc9-ccf98d5932e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:14.217792 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:14.217707 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:14.566540 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:14.566372 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d"] Jul 24 23:26:14.569284 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:26:14.569258 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c154516_0fb2_407e_bdc9_ccf98d5932e3.slice/crio-7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29 WatchSource:0}: Error finding container 7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29: Status 404 returned error can't find the container with id 7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29 Jul 24 23:26:15.369903 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:15.369852 2571 generic.go:356] "Generic (PLEG): container finished" podID="2c154516-0fb2-407e-bdc9-ccf98d5932e3" containerID="4165c71cec61b2fe7f98b79f0280769e71d9e86c00551f47dbd74b9b63204732" exitCode=0 Jul 24 23:26:15.370279 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:15.369948 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" event={"ID":"2c154516-0fb2-407e-bdc9-ccf98d5932e3","Type":"ContainerDied","Data":"4165c71cec61b2fe7f98b79f0280769e71d9e86c00551f47dbd74b9b63204732"} Jul 24 23:26:15.370279 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:15.370002 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" event={"ID":"2c154516-0fb2-407e-bdc9-ccf98d5932e3","Type":"ContainerStarted","Data":"7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29"} Jul 24 23:26:16.512008 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:16.511980 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:16.540161 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:16.540126 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca\") pod \"2c154516-0fb2-407e-bdc9-ccf98d5932e3\" (UID: \"2c154516-0fb2-407e-bdc9-ccf98d5932e3\") " Jul 24 23:26:16.540488 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:16.540465 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2c154516-0fb2-407e-bdc9-ccf98d5932e3" (UID: "2c154516-0fb2-407e-bdc9-ccf98d5932e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:26:16.641354 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:16.641268 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c154516-0fb2-407e-bdc9-ccf98d5932e3-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:26:17.381328 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:17.381297 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" Jul 24 23:26:17.381542 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:17.381289 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d" event={"ID":"2c154516-0fb2-407e-bdc9-ccf98d5932e3","Type":"ContainerDied","Data":"7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29"} Jul 24 23:26:17.381542 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:17.381482 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f504e461a730d81e8d64bad2f7f4caeb156ae97b98555676b32f225b7c99e29" Jul 24 23:26:33.979920 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:33.979871 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d"] Jul 24 23:26:33.981625 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:33.981603 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4ab9b5-vm82d"] Jul 24 23:26:35.560521 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:26:35.560427 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c154516-0fb2-407e-bdc9-ccf98d5932e3" path="/var/lib/kubelet/pods/2c154516-0fb2-407e-bdc9-ccf98d5932e3/volumes" Jul 24 23:27:36.453505 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.453468 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:27:36.454100 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.454084 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c154516-0fb2-407e-bdc9-ccf98d5932e3" containerName="revoke-keys" Jul 24 23:27:36.454152 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.454103 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c154516-0fb2-407e-bdc9-ccf98d5932e3" containerName="revoke-keys" Jul 24 23:27:36.454199 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.454189 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c154516-0fb2-407e-bdc9-ccf98d5932e3" containerName="revoke-keys" Jul 24 23:27:36.457700 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.457682 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.459522 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.459501 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"kube-root-ca.crt\"" Jul 24 23:27:36.460043 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.459987 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"auth-test-model-5c850d-kserve-self-signed-certs\"" Jul 24 23:27:36.460165 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.460055 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"openshift-service-ca.crt\"" Jul 24 23:27:36.460255 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.460230 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"default-dockercfg-g4tk7\"" Jul 24 23:27:36.463712 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.463689 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:27:36.528027 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.527994 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.528190 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.528056 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.528190 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.528122 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.528281 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.528216 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmshp\" (UniqueName: \"kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.528281 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.528236 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.528281 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.528265 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.628847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.628810 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmshp\" (UniqueName: \"kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.628847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.628848 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629093 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.628876 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629093 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.628919 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629093 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.628994 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629093 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.629029 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629326 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.629294 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629413 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.629342 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.629485 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.629436 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.631124 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.631103 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.631439 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.631420 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.635837 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.635815 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmshp\" (UniqueName: \"kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp\") pod \"auth-test-model-5c850d-kserve-5c7cc4799f-24dl6\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.768319 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.768232 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:36.901911 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:36.901888 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:27:36.904718 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:27:36.904683 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4281e38e_57c0_4e96_9ce8_70b3a7241061.slice/crio-f7933eb17eb1aef0b0c447998e383661cdf59797525857acc47b73ca5f59400d WatchSource:0}: Error finding container f7933eb17eb1aef0b0c447998e383661cdf59797525857acc47b73ca5f59400d: Status 404 returned error can't find the container with id f7933eb17eb1aef0b0c447998e383661cdf59797525857acc47b73ca5f59400d Jul 24 23:27:37.791470 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:37.791426 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" event={"ID":"4281e38e-57c0-4e96-9ce8-70b3a7241061","Type":"ContainerStarted","Data":"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d"} Jul 24 23:27:37.791470 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:37.791475 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" event={"ID":"4281e38e-57c0-4e96-9ce8-70b3a7241061","Type":"ContainerStarted","Data":"f7933eb17eb1aef0b0c447998e383661cdf59797525857acc47b73ca5f59400d"} Jul 24 23:27:37.807323 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:37.807269 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" podStartSLOduration=1.807255236 podStartE2EDuration="1.807255236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:37.806286328 +0000 UTC m=+2486.735625538" watchObservedRunningTime="2026-07-24 23:27:37.807255236 +0000 UTC m=+2486.736594448" Jul 24 23:27:46.769080 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:46.769041 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:46.769080 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:46.769085 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:46.781820 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:46.781797 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:46.852089 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:46.852062 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:27:56.153443 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.153408 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:27:56.157111 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.157088 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.158968 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.158944 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"kube-root-ca.crt\"" Jul 24 23:27:56.159337 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.159317 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"openshift-service-ca.crt\"" Jul 24 23:27:56.159444 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.159363 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"default-dockercfg-wg8d9\"" Jul 24 23:27:56.159444 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.159424 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"auth-test-model-d08a26-kserve-self-signed-certs\"" Jul 24 23:27:56.167304 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.167281 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:27:56.217839 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217803 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.217839 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217841 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vvc\" (UniqueName: \"kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.218042 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217863 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.218042 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217920 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.218042 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217964 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.218042 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.217996 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319265 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319229 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319333 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319372 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vvc\" (UniqueName: \"kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319433 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319646 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319489 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319646 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319578 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319742 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319670 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319904 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319885 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.319981 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.319926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.321742 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.321718 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.321906 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.321842 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.329965 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.329940 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vvc\" (UniqueName: \"kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc\") pod \"auth-test-model-d08a26-kserve-86cbbcb879-pml5m\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.467726 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.467638 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:27:56.603775 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.603643 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:27:56.606769 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:27:56.606738 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96b8bd03_5160_445d_930e_82d28632ee40.slice/crio-024952379d7f8ae89d81ddd4cf6bca87cc91f696616a31bb64680152a05d971c WatchSource:0}: Error finding container 024952379d7f8ae89d81ddd4cf6bca87cc91f696616a31bb64680152a05d971c: Status 404 returned error can't find the container with id 024952379d7f8ae89d81ddd4cf6bca87cc91f696616a31bb64680152a05d971c Jul 24 23:27:56.891104 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.891063 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" event={"ID":"96b8bd03-5160-445d-930e-82d28632ee40","Type":"ContainerStarted","Data":"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4"} Jul 24 23:27:56.891285 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.891116 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" event={"ID":"96b8bd03-5160-445d-930e-82d28632ee40","Type":"ContainerStarted","Data":"024952379d7f8ae89d81ddd4cf6bca87cc91f696616a31bb64680152a05d971c"} Jul 24 23:27:56.906831 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:27:56.906779 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" podStartSLOduration=0.906763257 podStartE2EDuration="906.763257ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:27:56.904636208 +0000 UTC m=+2505.833975421" watchObservedRunningTime="2026-07-24 23:27:56.906763257 +0000 UTC m=+2505.836102469" Jul 24 23:28:06.468306 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:28:06.468219 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:28:06.468306 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:28:06.468276 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:28:06.480784 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:28:06.480755 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:28:06.953871 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:28:06.953840 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:30:03.563234 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:03.563157 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:30:04.569386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:04.569350 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:30:18.584095 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.584033 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" podUID="4281e38e-57c0-4e96-9ce8-70b3a7241061" containerName="main" containerID="cri-o://8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d" gracePeriod=45 Jul 24 23:30:18.844173 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.844119 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:30:18.916154 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916119 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916154 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916157 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916433 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916176 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmshp\" (UniqueName: \"kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916433 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916198 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916433 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916224 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916433 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916274 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir\") pod \"4281e38e-57c0-4e96-9ce8-70b3a7241061\" (UID: \"4281e38e-57c0-4e96-9ce8-70b3a7241061\") " Jul 24 23:30:18.916666 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916515 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home" (OuterVolumeSpecName: "home") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:18.916666 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916551 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache" (OuterVolumeSpecName: "model-cache") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:18.916666 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916631 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:18.916823 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916668 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:18.916823 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.916680 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:18.918386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.918365 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm" (OuterVolumeSpecName: "dshm") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:18.918386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.918374 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:30:18.918542 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:18.918479 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp" (OuterVolumeSpecName: "kube-api-access-zmshp") pod "4281e38e-57c0-4e96-9ce8-70b3a7241061" (UID: "4281e38e-57c0-4e96-9ce8-70b3a7241061"). InnerVolumeSpecName "kube-api-access-zmshp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:19.018040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.018001 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmshp\" (UniqueName: \"kubernetes.io/projected/4281e38e-57c0-4e96-9ce8-70b3a7241061-kube-api-access-zmshp\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.018040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.018033 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4281e38e-57c0-4e96-9ce8-70b3a7241061-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.018040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.018042 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.018040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.018049 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4281e38e-57c0-4e96-9ce8-70b3a7241061-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.587972 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.587932 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" podUID="96b8bd03-5160-445d-930e-82d28632ee40" containerName="main" containerID="cri-o://db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4" gracePeriod=45 Jul 24 23:30:19.611715 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.611681 2571 generic.go:356] "Generic (PLEG): container finished" podID="4281e38e-57c0-4e96-9ce8-70b3a7241061" containerID="8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d" exitCode=0 Jul 24 23:30:19.611847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.611729 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" Jul 24 23:30:19.611847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.611763 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" event={"ID":"4281e38e-57c0-4e96-9ce8-70b3a7241061","Type":"ContainerDied","Data":"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d"} Jul 24 23:30:19.611847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.611807 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6" event={"ID":"4281e38e-57c0-4e96-9ce8-70b3a7241061","Type":"ContainerDied","Data":"f7933eb17eb1aef0b0c447998e383661cdf59797525857acc47b73ca5f59400d"} Jul 24 23:30:19.611847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.611836 2571 scope.go:117] "RemoveContainer" containerID="8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d" Jul 24 23:30:19.656654 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.656616 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:30:19.660381 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.660352 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/auth-test-model-5c850d-kserve-5c7cc4799f-24dl6"] Jul 24 23:30:19.698296 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.698276 2571 scope.go:117] "RemoveContainer" containerID="8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d" Jul 24 23:30:19.698640 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:30:19.698620 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d\": container with ID starting with 8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d not found: ID does not exist" containerID="8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d" Jul 24 23:30:19.698719 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.698650 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d"} err="failed to get container status \"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d\": rpc error: code = NotFound desc = could not find container \"8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d\": container with ID starting with 8e12deec1ff1f2186f07396c7b81a1a8f046447aead34ecce9c6b2b1d86d2a3d not found: ID does not exist" Jul 24 23:30:19.847029 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.847005 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:30:19.927997 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.927961 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928165 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928030 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928165 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928063 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928165 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928088 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928165 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928138 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8vvc\" (UniqueName: \"kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928188 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir\") pod \"96b8bd03-5160-445d-930e-82d28632ee40\" (UID: \"96b8bd03-5160-445d-930e-82d28632ee40\") " Jul 24 23:30:19.928383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928316 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home" (OuterVolumeSpecName: "home") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:19.928530 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928436 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache" (OuterVolumeSpecName: "model-cache") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:19.928530 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928515 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:19.928661 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928645 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.928706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928667 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.928706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.928682 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:19.930264 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.930241 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc" (OuterVolumeSpecName: "kube-api-access-z8vvc") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "kube-api-access-z8vvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:19.930357 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.930307 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm" (OuterVolumeSpecName: "dshm") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:19.930357 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:19.930308 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "96b8bd03-5160-445d-930e-82d28632ee40" (UID: "96b8bd03-5160-445d-930e-82d28632ee40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:30:20.030168 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.030130 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96b8bd03-5160-445d-930e-82d28632ee40-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:20.030168 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.030164 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8vvc\" (UniqueName: \"kubernetes.io/projected/96b8bd03-5160-445d-930e-82d28632ee40-kube-api-access-z8vvc\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:20.030168 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.030173 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96b8bd03-5160-445d-930e-82d28632ee40-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:30:20.619175 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.619136 2571 generic.go:356] "Generic (PLEG): container finished" podID="96b8bd03-5160-445d-930e-82d28632ee40" containerID="db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4" exitCode=0 Jul 24 23:30:20.619175 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.619179 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" Jul 24 23:30:20.619749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.619195 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" event={"ID":"96b8bd03-5160-445d-930e-82d28632ee40","Type":"ContainerDied","Data":"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4"} Jul 24 23:30:20.619749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.619225 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m" event={"ID":"96b8bd03-5160-445d-930e-82d28632ee40","Type":"ContainerDied","Data":"024952379d7f8ae89d81ddd4cf6bca87cc91f696616a31bb64680152a05d971c"} Jul 24 23:30:20.619749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.619242 2571 scope.go:117] "RemoveContainer" containerID="db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4" Jul 24 23:30:20.628309 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.628290 2571 scope.go:117] "RemoveContainer" containerID="db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4" Jul 24 23:30:20.628596 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:30:20.628571 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4\": container with ID starting with db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4 not found: ID does not exist" containerID="db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4" Jul 24 23:30:20.628654 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.628603 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4"} err="failed to get container status \"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4\": rpc error: code = NotFound desc = could not find container \"db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4\": container with ID starting with db0977233abec21eb4d34805421910834dbb39295937215bdf22743e57fe61c4 not found: ID does not exist" Jul 24 23:30:20.638596 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.638573 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:30:20.641798 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:20.641776 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/auth-test-model-d08a26-kserve-86cbbcb879-pml5m"] Jul 24 23:30:21.559780 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:21.559745 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4281e38e-57c0-4e96-9ce8-70b3a7241061" path="/var/lib/kubelet/pods/4281e38e-57c0-4e96-9ce8-70b3a7241061/volumes" Jul 24 23:30:21.560189 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:21.560174 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b8bd03-5160-445d-930e-82d28632ee40" path="/var/lib/kubelet/pods/96b8bd03-5160-445d-930e-82d28632ee40/volumes" Jul 24 23:30:37.255808 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.255760 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:30:37.256289 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256231 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4281e38e-57c0-4e96-9ce8-70b3a7241061" containerName="main" Jul 24 23:30:37.256289 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256244 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4281e38e-57c0-4e96-9ce8-70b3a7241061" containerName="main" Jul 24 23:30:37.256289 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256262 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b8bd03-5160-445d-930e-82d28632ee40" containerName="main" Jul 24 23:30:37.256289 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256270 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b8bd03-5160-445d-930e-82d28632ee40" containerName="main" Jul 24 23:30:37.256448 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256333 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4281e38e-57c0-4e96-9ce8-70b3a7241061" containerName="main" Jul 24 23:30:37.256448 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.256347 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b8bd03-5160-445d-930e-82d28632ee40" containerName="main" Jul 24 23:30:37.261318 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.261298 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.263713 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.263678 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"kube-root-ca.crt\"" Jul 24 23:30:37.263848 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.263722 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"default-dockercfg-wg8d9\"" Jul 24 23:30:37.263848 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.263685 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"openshift-service-ca.crt\"" Jul 24 23:30:37.263848 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.263795 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d08a26\"/\"sub-test-model-d08a26-kserve-self-signed-certs\"" Jul 24 23:30:37.267439 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.267382 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:30:37.394030 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.393990 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.394030 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.394036 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.394243 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.394067 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.394243 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.394135 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84zdg\" (UniqueName: \"kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.394316 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.394269 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.394350 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.394308 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495019 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.494984 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495067 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495088 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495109 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495137 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84zdg\" (UniqueName: \"kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495500 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495466 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495570 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495515 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.495570 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.495559 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.497357 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.497335 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.497562 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.497546 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.502360 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.502339 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84zdg\" (UniqueName: \"kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg\") pod \"sub-test-model-d08a26-kserve-7bd5f97777-rkvvr\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.571706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.571670 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:37.700932 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.700910 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:30:37.703721 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:30:37.703695 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd86104cb_75f6_475a_9e50_482feb198164.slice/crio-8177578c767678a226700b649b6c243156c64c203cb19658cd487bfd26a8aa7f WatchSource:0}: Error finding container 8177578c767678a226700b649b6c243156c64c203cb19658cd487bfd26a8aa7f: Status 404 returned error can't find the container with id 8177578c767678a226700b649b6c243156c64c203cb19658cd487bfd26a8aa7f Jul 24 23:30:37.714896 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:37.714870 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" event={"ID":"d86104cb-75f6-475a-9e50-482feb198164","Type":"ContainerStarted","Data":"8177578c767678a226700b649b6c243156c64c203cb19658cd487bfd26a8aa7f"} Jul 24 23:30:38.721440 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:38.721389 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" event={"ID":"d86104cb-75f6-475a-9e50-482feb198164","Type":"ContainerStarted","Data":"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc"} Jul 24 23:30:38.735041 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:38.734971 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" podStartSLOduration=1.7349529430000001 podStartE2EDuration="1.734952943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:38.734333281 +0000 UTC m=+2667.663672495" watchObservedRunningTime="2026-07-24 23:30:38.734952943 +0000 UTC m=+2667.664292154" Jul 24 23:30:47.572176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:47.572138 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:47.574812 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:47.572189 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:47.584591 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:47.584568 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:30:47.779930 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:30:47.779901 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:31:12.566312 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:12.566283 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:31:12.571602 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:12.571576 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:31:12.573982 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:12.573959 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:31:12.579154 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:12.579136 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:31:12.597194 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:12.597171 2571 scope.go:117] "RemoveContainer" containerID="0f1f1ea8f7f8e78357e64a560969ca0c0cb6b39bc36fe88eafc48157c7473e4a" Jul 24 23:31:22.281113 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:22.281079 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:31:31.561415 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.561367 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:31:31.565163 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.565142 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.567583 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.567545 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"rate-test-model-5c850d-kserve-self-signed-certs\"" Jul 24 23:31:31.567583 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.567555 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"kube-root-ca.crt\"" Jul 24 23:31:31.567783 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.567594 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"default-dockercfg-g4tk7\"" Jul 24 23:31:31.567783 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.567612 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5c850d\"/\"openshift-service-ca.crt\"" Jul 24 23:31:31.572052 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.572030 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:31:31.627668 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627630 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.627668 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627667 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.627875 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627701 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.627875 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627788 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxqls\" (UniqueName: \"kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.627875 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627809 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.627875 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.627845 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728357 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728314 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728593 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728440 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxqls\" (UniqueName: \"kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728593 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728477 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728593 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728522 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728593 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728578 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.728809 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728599 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.729004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728976 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.729081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.728992 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.729081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.729028 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.730753 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.730731 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.731051 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.731030 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.735236 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.735213 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxqls\" (UniqueName: \"kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls\") pod \"rate-test-model-5c850d-kserve-669989d5c5-zqglc\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:31.875650 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:31.875554 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:32.015323 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:32.015298 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:31:32.017846 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:31:32.017821 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8249ea69_7c50_4989_af13_c4195c8e5c4f.slice/crio-8212071e4fabc2f9b44c18685d0c3f5a077e3065f1f6fef264555f73a640c3fa WatchSource:0}: Error finding container 8212071e4fabc2f9b44c18685d0c3f5a077e3065f1f6fef264555f73a640c3fa: Status 404 returned error can't find the container with id 8212071e4fabc2f9b44c18685d0c3f5a077e3065f1f6fef264555f73a640c3fa Jul 24 23:31:32.995194 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:32.995156 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" event={"ID":"8249ea69-7c50-4989-af13-c4195c8e5c4f","Type":"ContainerStarted","Data":"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80"} Jul 24 23:31:32.995194 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:32.995203 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" event={"ID":"8249ea69-7c50-4989-af13-c4195c8e5c4f","Type":"ContainerStarted","Data":"8212071e4fabc2f9b44c18685d0c3f5a077e3065f1f6fef264555f73a640c3fa"} Jul 24 23:31:33.011084 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:33.011018 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" podStartSLOduration=2.011002861 podStartE2EDuration="2.011002861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:31:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:31:33.008638324 +0000 UTC m=+2721.937977535" watchObservedRunningTime="2026-07-24 23:31:33.011002861 +0000 UTC m=+2721.940342072" Jul 24 23:31:37.300376 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.300331 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" podUID="d86104cb-75f6-475a-9e50-482feb198164" containerName="main" containerID="cri-o://f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc" gracePeriod=45 Jul 24 23:31:37.556890 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.556817 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:31:37.687658 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687624 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84zdg\" (UniqueName: \"kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.687851 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687672 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.687851 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687751 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.687851 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687808 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.688025 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687858 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.688025 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.687889 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir\") pod \"d86104cb-75f6-475a-9e50-482feb198164\" (UID: \"d86104cb-75f6-475a-9e50-482feb198164\") " Jul 24 23:31:37.688126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.688046 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home" (OuterVolumeSpecName: "home") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:37.688126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.688077 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache" (OuterVolumeSpecName: "model-cache") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:37.688258 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.688230 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:37.688311 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.688256 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:37.688311 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.688276 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:37.689875 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.689846 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm" (OuterVolumeSpecName: "dshm") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:31:37.689989 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.689880 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:31:37.689989 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.689884 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg" (OuterVolumeSpecName: "kube-api-access-84zdg") pod "d86104cb-75f6-475a-9e50-482feb198164" (UID: "d86104cb-75f6-475a-9e50-482feb198164"). InnerVolumeSpecName "kube-api-access-84zdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:31:37.789126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.789096 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:37.789126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.789123 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d86104cb-75f6-475a-9e50-482feb198164-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:37.789126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.789133 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d86104cb-75f6-475a-9e50-482feb198164-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:37.789335 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:37.789141 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84zdg\" (UniqueName: \"kubernetes.io/projected/d86104cb-75f6-475a-9e50-482feb198164-kube-api-access-84zdg\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:31:38.020589 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.020553 2571 generic.go:356] "Generic (PLEG): container finished" podID="d86104cb-75f6-475a-9e50-482feb198164" containerID="f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc" exitCode=0 Jul 24 23:31:38.020749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.020595 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" Jul 24 23:31:38.020749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.020598 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" event={"ID":"d86104cb-75f6-475a-9e50-482feb198164","Type":"ContainerDied","Data":"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc"} Jul 24 23:31:38.020749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.020720 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr" event={"ID":"d86104cb-75f6-475a-9e50-482feb198164","Type":"ContainerDied","Data":"8177578c767678a226700b649b6c243156c64c203cb19658cd487bfd26a8aa7f"} Jul 24 23:31:38.020749 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.020740 2571 scope.go:117] "RemoveContainer" containerID="f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc" Jul 24 23:31:38.032529 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.032503 2571 scope.go:117] "RemoveContainer" containerID="f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc" Jul 24 23:31:38.033105 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:31:38.033066 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc\": container with ID starting with f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc not found: ID does not exist" containerID="f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc" Jul 24 23:31:38.033207 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.033110 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc"} err="failed to get container status \"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc\": rpc error: code = NotFound desc = could not find container \"f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc\": container with ID starting with f7e7b71f498f41d6c5bb99d76a75f39587f2be13a7a700e3e007533f5e2054fc not found: ID does not exist" Jul 24 23:31:38.041680 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.041657 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:31:38.044845 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:38.044824 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d08a26/sub-test-model-d08a26-kserve-7bd5f97777-rkvvr"] Jul 24 23:31:39.560659 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:39.560622 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d86104cb-75f6-475a-9e50-482feb198164" path="/var/lib/kubelet/pods/d86104cb-75f6-475a-9e50-482feb198164/volumes" Jul 24 23:31:41.875995 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:41.875936 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:41.875995 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:41.876008 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:41.889542 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:41.889508 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:31:42.053211 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:31:42.053174 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:32:49.572140 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:49.572057 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:32:51.413133 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.413100 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698612756144"] Jul 24 23:32:51.413543 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.413532 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d86104cb-75f6-475a-9e50-482feb198164" containerName="main" Jul 24 23:32:51.413543 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.413544 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d86104cb-75f6-475a-9e50-482feb198164" containerName="main" Jul 24 23:32:51.413633 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.413622 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d86104cb-75f6-475a-9e50-482feb198164" containerName="main" Jul 24 23:32:51.416646 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.416628 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:51.418481 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.418451 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-q4xpp\"" Jul 24 23:32:51.420931 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.420909 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612756144"] Jul 24 23:32:51.507305 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.507267 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m7kj\" (UniqueName: \"kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj\") pod \"test-curl-7613-139698612756144\" (UID: \"14705c79-6fbb-4e75-a82f-ed056e3a4777\") " pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:51.608135 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.608095 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m7kj\" (UniqueName: \"kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj\") pod \"test-curl-7613-139698612756144\" (UID: \"14705c79-6fbb-4e75-a82f-ed056e3a4777\") " pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:51.616647 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.616619 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m7kj\" (UniqueName: \"kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj\") pod \"test-curl-7613-139698612756144\" (UID: \"14705c79-6fbb-4e75-a82f-ed056e3a4777\") " pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:51.727905 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.727812 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:51.860203 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.860179 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612756144"] Jul 24 23:32:51.862042 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:32:51.862016 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14705c79_6fbb_4e75_a82f_ed056e3a4777.slice/crio-5f603b69a1cce3bb483bdf1434dde45356a5076d168f13a7eeca2174e4fda4a3 WatchSource:0}: Error finding container 5f603b69a1cce3bb483bdf1434dde45356a5076d168f13a7eeca2174e4fda4a3: Status 404 returned error can't find the container with id 5f603b69a1cce3bb483bdf1434dde45356a5076d168f13a7eeca2174e4fda4a3 Jul 24 23:32:51.863357 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:51.863337 2571 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:32:52.399378 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:52.399334 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612756144" event={"ID":"14705c79-6fbb-4e75-a82f-ed056e3a4777","Type":"ContainerStarted","Data":"5f603b69a1cce3bb483bdf1434dde45356a5076d168f13a7eeca2174e4fda4a3"} Jul 24 23:32:53.406137 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:53.406097 2571 generic.go:356] "Generic (PLEG): container finished" podID="14705c79-6fbb-4e75-a82f-ed056e3a4777" containerID="321cd183782284c2ec9a16cd1ad8e7dacb366a7d244040a8823756ad40b11bb5" exitCode=0 Jul 24 23:32:53.406574 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:53.406185 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612756144" event={"ID":"14705c79-6fbb-4e75-a82f-ed056e3a4777","Type":"ContainerDied","Data":"321cd183782284c2ec9a16cd1ad8e7dacb366a7d244040a8823756ad40b11bb5"} Jul 24 23:32:54.548798 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.548774 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:54.582385 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.582352 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698612756144_14705c79-6fbb-4e75-a82f-ed056e3a4777/test-curl-7613-139698612756144/0.log" Jul 24 23:32:54.605523 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.605486 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698612756144"] Jul 24 23:32:54.609208 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.609183 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698612756144"] Jul 24 23:32:54.640125 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.640092 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2m7kj\" (UniqueName: \"kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj\") pod \"14705c79-6fbb-4e75-a82f-ed056e3a4777\" (UID: \"14705c79-6fbb-4e75-a82f-ed056e3a4777\") " Jul 24 23:32:54.642244 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.642219 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj" (OuterVolumeSpecName: "kube-api-access-2m7kj") pod "14705c79-6fbb-4e75-a82f-ed056e3a4777" (UID: "14705c79-6fbb-4e75-a82f-ed056e3a4777"). InnerVolumeSpecName "kube-api-access-2m7kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:32:54.725620 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.725538 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698612756272"] Jul 24 23:32:54.725981 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.725966 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14705c79-6fbb-4e75-a82f-ed056e3a4777" containerName="test-curl-7613-139698612756144" Jul 24 23:32:54.726028 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.725985 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="14705c79-6fbb-4e75-a82f-ed056e3a4777" containerName="test-curl-7613-139698612756144" Jul 24 23:32:54.726065 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.726048 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="14705c79-6fbb-4e75-a82f-ed056e3a4777" containerName="test-curl-7613-139698612756144" Jul 24 23:32:54.729241 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.729222 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:54.734968 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.734941 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612756272"] Jul 24 23:32:54.741859 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.741831 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m7kj\" (UniqueName: \"kubernetes.io/projected/14705c79-6fbb-4e75-a82f-ed056e3a4777-kube-api-access-2m7kj\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:32:54.843240 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.843199 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4c7s\" (UniqueName: \"kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s\") pod \"test-curl-7613-139698612756272\" (UID: \"d8d6bb53-d5c4-4f47-956c-e297acbc456d\") " pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:54.944068 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.944032 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4c7s\" (UniqueName: \"kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s\") pod \"test-curl-7613-139698612756272\" (UID: \"d8d6bb53-d5c4-4f47-956c-e297acbc456d\") " pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:54.951209 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:54.951177 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4c7s\" (UniqueName: \"kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s\") pod \"test-curl-7613-139698612756272\" (UID: \"d8d6bb53-d5c4-4f47-956c-e297acbc456d\") " pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:55.040415 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.040361 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:55.177147 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.177122 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612756272"] Jul 24 23:32:55.179325 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:32:55.179296 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8d6bb53_d5c4_4f47_956c_e297acbc456d.slice/crio-faa999a1a92fe9385459f127976622334a77df1242e9012b83a252679ff227b7 WatchSource:0}: Error finding container faa999a1a92fe9385459f127976622334a77df1242e9012b83a252679ff227b7: Status 404 returned error can't find the container with id faa999a1a92fe9385459f127976622334a77df1242e9012b83a252679ff227b7 Jul 24 23:32:55.417341 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.417238 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612756272" event={"ID":"d8d6bb53-d5c4-4f47-956c-e297acbc456d","Type":"ContainerStarted","Data":"faa999a1a92fe9385459f127976622334a77df1242e9012b83a252679ff227b7"} Jul 24 23:32:55.418367 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.418344 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f603b69a1cce3bb483bdf1434dde45356a5076d168f13a7eeca2174e4fda4a3" Jul 24 23:32:55.418367 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.418353 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:55.428726 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:32:55.428693 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698612756144\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="14705c79-6fbb-4e75-a82f-ed056e3a4777" pod="opendatahub/test-curl-7613-139698612756144" Jul 24 23:32:55.560483 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:55.560451 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14705c79-6fbb-4e75-a82f-ed056e3a4777" path="/var/lib/kubelet/pods/14705c79-6fbb-4e75-a82f-ed056e3a4777/volumes" Jul 24 23:32:56.425411 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:56.425361 2571 generic.go:356] "Generic (PLEG): container finished" podID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" containerID="247effbe790c2740c5247f68ff1cd2ca3e72bef3fee001dfdf86ba3cbec3b49d" exitCode=0 Jul 24 23:32:56.425589 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:56.425447 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612756272" event={"ID":"d8d6bb53-d5c4-4f47-956c-e297acbc456d","Type":"ContainerDied","Data":"247effbe790c2740c5247f68ff1cd2ca3e72bef3fee001dfdf86ba3cbec3b49d"} Jul 24 23:32:57.566847 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.566821 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:57.585452 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.585422 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698612756272_d8d6bb53-d5c4-4f47-956c-e297acbc456d/test-curl-7613-139698612756272/0.log" Jul 24 23:32:57.608942 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.608914 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698612756272"] Jul 24 23:32:57.612477 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.612451 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698612756272"] Jul 24 23:32:57.673512 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.673482 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r4c7s\" (UniqueName: \"kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s\") pod \"d8d6bb53-d5c4-4f47-956c-e297acbc456d\" (UID: \"d8d6bb53-d5c4-4f47-956c-e297acbc456d\") " Jul 24 23:32:57.675605 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.675570 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s" (OuterVolumeSpecName: "kube-api-access-r4c7s") pod "d8d6bb53-d5c4-4f47-956c-e297acbc456d" (UID: "d8d6bb53-d5c4-4f47-956c-e297acbc456d"). InnerVolumeSpecName "kube-api-access-r4c7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:32:57.771234 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.771199 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:32:57.771631 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.771610 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" containerName="test-curl-7613-139698612756272" Jul 24 23:32:57.771631 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.771627 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" containerName="test-curl-7613-139698612756272" Jul 24 23:32:57.771787 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.771711 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" containerName="test-curl-7613-139698612756272" Jul 24 23:32:57.774401 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.774372 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4c7s\" (UniqueName: \"kubernetes.io/projected/d8d6bb53-d5c4-4f47-956c-e297acbc456d-kube-api-access-r4c7s\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:32:57.774706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.774692 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:32:57.779690 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.779664 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:32:57.875933 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.875891 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxl9\" (UniqueName: \"kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9\") pod \"test-curl-7613-139698616268528\" (UID: \"3345b175-07bd-4c84-95a3-1139c816af55\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:32:57.977531 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.977419 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxl9\" (UniqueName: \"kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9\") pod \"test-curl-7613-139698616268528\" (UID: \"3345b175-07bd-4c84-95a3-1139c816af55\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:32:57.984790 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:57.984756 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxl9\" (UniqueName: \"kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9\") pod \"test-curl-7613-139698616268528\" (UID: \"3345b175-07bd-4c84-95a3-1139c816af55\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:32:58.085146 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:58.085107 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:32:58.213981 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:58.213947 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:32:58.216588 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:32:58.216556 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3345b175_07bd_4c84_95a3_1139c816af55.slice/crio-4f1e0fa307fe85f7b9908a0617f92492e2c0d7b3d2bf017c0178208f8a8ba238 WatchSource:0}: Error finding container 4f1e0fa307fe85f7b9908a0617f92492e2c0d7b3d2bf017c0178208f8a8ba238: Status 404 returned error can't find the container with id 4f1e0fa307fe85f7b9908a0617f92492e2c0d7b3d2bf017c0178208f8a8ba238 Jul 24 23:32:58.437585 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:58.437558 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="faa999a1a92fe9385459f127976622334a77df1242e9012b83a252679ff227b7" Jul 24 23:32:58.437585 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:58.437562 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:58.438865 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:58.438837 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698616268528" event={"ID":"3345b175-07bd-4c84-95a3-1139c816af55","Type":"ContainerStarted","Data":"4f1e0fa307fe85f7b9908a0617f92492e2c0d7b3d2bf017c0178208f8a8ba238"} Jul 24 23:32:58.447461 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:32:58.447435 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698612756272\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:59.445456 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:59.445418 2571 generic.go:356] "Generic (PLEG): container finished" podID="3345b175-07bd-4c84-95a3-1139c816af55" containerID="30e3645c155c7bb8f8351c274b8d9724713e0dcd16d682c371c238b370e844e6" exitCode=0 Jul 24 23:32:59.445849 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:59.445498 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698616268528" event={"ID":"3345b175-07bd-4c84-95a3-1139c816af55","Type":"ContainerDied","Data":"30e3645c155c7bb8f8351c274b8d9724713e0dcd16d682c371c238b370e844e6"} Jul 24 23:32:59.447024 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:32:59.446998 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698612756272\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" pod="opendatahub/test-curl-7613-139698612756272" Jul 24 23:32:59.560261 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:32:59.560224 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8d6bb53-d5c4-4f47-956c-e297acbc456d" path="/var/lib/kubelet/pods/d8d6bb53-d5c4-4f47-956c-e297acbc456d/volumes" Jul 24 23:33:00.591181 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.591151 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:00.609923 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.609875 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698616268528_3345b175-07bd-4c84-95a3-1139c816af55/test-curl-7613-139698616268528/0.log" Jul 24 23:33:00.634533 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.634489 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:00.639783 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.639749 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:00.703665 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.703577 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgxl9\" (UniqueName: \"kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9\") pod \"3345b175-07bd-4c84-95a3-1139c816af55\" (UID: \"3345b175-07bd-4c84-95a3-1139c816af55\") " Jul 24 23:33:00.705605 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.705580 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9" (OuterVolumeSpecName: "kube-api-access-rgxl9") pod "3345b175-07bd-4c84-95a3-1139c816af55" (UID: "3345b175-07bd-4c84-95a3-1139c816af55"). InnerVolumeSpecName "kube-api-access-rgxl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:33:00.751552 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.751517 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:00.752013 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.751994 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3345b175-07bd-4c84-95a3-1139c816af55" containerName="test-curl-7613-139698616268528" Jul 24 23:33:00.752107 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.752015 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3345b175-07bd-4c84-95a3-1139c816af55" containerName="test-curl-7613-139698616268528" Jul 24 23:33:00.752178 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.752163 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3345b175-07bd-4c84-95a3-1139c816af55" containerName="test-curl-7613-139698616268528" Jul 24 23:33:00.755352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.755331 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:00.760173 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.760147 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:00.805502 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.805464 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxl9\" (UniqueName: \"kubernetes.io/projected/3345b175-07bd-4c84-95a3-1139c816af55-kube-api-access-rgxl9\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:00.906547 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:00.906507 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxrl\" (UniqueName: \"kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl\") pod \"test-curl-7613-139698616268528\" (UID: \"4ea35d58-6be0-42b5-bed4-0a8863c10bed\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:01.008038 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.007955 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxrl\" (UniqueName: \"kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl\") pod \"test-curl-7613-139698616268528\" (UID: \"4ea35d58-6be0-42b5-bed4-0a8863c10bed\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:01.015153 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.015118 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxrl\" (UniqueName: \"kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl\") pod \"test-curl-7613-139698616268528\" (UID: \"4ea35d58-6be0-42b5-bed4-0a8863c10bed\") " pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:01.065408 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.065356 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:01.200117 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.200092 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:01.202371 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:33:01.202344 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea35d58_6be0_42b5_bed4_0a8863c10bed.slice/crio-6445e31f5338ab00ae00169c1b781d1a60f739a2cbbe50f6ee069600c4e34df6 WatchSource:0}: Error finding container 6445e31f5338ab00ae00169c1b781d1a60f739a2cbbe50f6ee069600c4e34df6: Status 404 returned error can't find the container with id 6445e31f5338ab00ae00169c1b781d1a60f739a2cbbe50f6ee069600c4e34df6 Jul 24 23:33:01.456739 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.456692 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698616268528" event={"ID":"4ea35d58-6be0-42b5-bed4-0a8863c10bed","Type":"ContainerStarted","Data":"6445e31f5338ab00ae00169c1b781d1a60f739a2cbbe50f6ee069600c4e34df6"} Jul 24 23:33:01.457945 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.457925 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f1e0fa307fe85f7b9908a0617f92492e2c0d7b3d2bf017c0178208f8a8ba238" Jul 24 23:33:01.457945 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.457924 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:01.469628 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.469595 2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7613-139698616268528" oldPodUID="3345b175-07bd-4c84-95a3-1139c816af55" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" Jul 24 23:33:01.563911 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:01.563876 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3345b175-07bd-4c84-95a3-1139c816af55" path="/var/lib/kubelet/pods/3345b175-07bd-4c84-95a3-1139c816af55/volumes" Jul 24 23:33:02.465104 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:02.465068 2571 generic.go:356] "Generic (PLEG): container finished" podID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" containerID="6f43186ddf1cc7318c92eaa47e14b578c96bb66055d1078d45a61ddcd775a9cd" exitCode=0 Jul 24 23:33:02.465518 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:02.465116 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698616268528" event={"ID":"4ea35d58-6be0-42b5-bed4-0a8863c10bed","Type":"ContainerDied","Data":"6f43186ddf1cc7318c92eaa47e14b578c96bb66055d1078d45a61ddcd775a9cd"} Jul 24 23:33:03.605039 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.605008 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:03.621995 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.621951 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698616268528_4ea35d58-6be0-42b5-bed4-0a8863c10bed/test-curl-7613-139698616268528/0.log" Jul 24 23:33:03.643211 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.643181 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:03.646583 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.646559 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698616268528"] Jul 24 23:33:03.732135 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.732027 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcxrl\" (UniqueName: \"kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl\") pod \"4ea35d58-6be0-42b5-bed4-0a8863c10bed\" (UID: \"4ea35d58-6be0-42b5-bed4-0a8863c10bed\") " Jul 24 23:33:03.734194 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.734170 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl" (OuterVolumeSpecName: "kube-api-access-dcxrl") pod "4ea35d58-6be0-42b5-bed4-0a8863c10bed" (UID: "4ea35d58-6be0-42b5-bed4-0a8863c10bed"). InnerVolumeSpecName "kube-api-access-dcxrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:33:03.767386 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.767351 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:03.767781 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.767766 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" containerName="test-curl-7613-139698616268528" Jul 24 23:33:03.767834 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.767782 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" containerName="test-curl-7613-139698616268528" Jul 24 23:33:03.767866 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.767843 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" containerName="test-curl-7613-139698616268528" Jul 24 23:33:03.771010 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.770993 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:03.774328 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.774301 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:03.833068 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.833030 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxrl\" (UniqueName: \"kubernetes.io/projected/4ea35d58-6be0-42b5-bed4-0a8863c10bed-kube-api-access-dcxrl\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:03.933717 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:03.933669 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtcxv\" (UniqueName: \"kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv\") pod \"test-curl-7613-139698612815312\" (UID: \"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:04.034976 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.034935 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtcxv\" (UniqueName: \"kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv\") pod \"test-curl-7613-139698612815312\" (UID: \"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:04.042918 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.042892 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtcxv\" (UniqueName: \"kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv\") pod \"test-curl-7613-139698612815312\" (UID: \"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:04.083017 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.082982 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:04.212596 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.212558 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:04.216132 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:33:04.216104 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod72166c1f_65d8_4b35_a975_e7cf4ff0c7ab.slice/crio-c56a0d5016065112d81017cfa1d5f125a8c26985d95f337224fde07256e4c43c WatchSource:0}: Error finding container c56a0d5016065112d81017cfa1d5f125a8c26985d95f337224fde07256e4c43c: Status 404 returned error can't find the container with id c56a0d5016065112d81017cfa1d5f125a8c26985d95f337224fde07256e4c43c Jul 24 23:33:04.477591 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.477507 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:04.477741 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.477520 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6445e31f5338ab00ae00169c1b781d1a60f739a2cbbe50f6ee069600c4e34df6" Jul 24 23:33:04.478778 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.478747 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612815312" event={"ID":"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab","Type":"ContainerStarted","Data":"c56a0d5016065112d81017cfa1d5f125a8c26985d95f337224fde07256e4c43c"} Jul 24 23:33:04.487706 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:04.487680 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698616268528\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:04.591262 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.591228 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" podUID="8249ea69-7c50-4989-af13-c4195c8e5c4f" containerName="main" containerID="cri-o://c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80" gracePeriod=45 Jul 24 23:33:04.845998 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.845973 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:33:04.847546 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:04.847520 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698616268528\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:04.943745 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943709 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.943745 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943749 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rxqls\" (UniqueName: \"kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.944006 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943775 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.944006 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943828 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.944006 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943908 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.944006 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.943948 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs\") pod \"8249ea69-7c50-4989-af13-c4195c8e5c4f\" (UID: \"8249ea69-7c50-4989-af13-c4195c8e5c4f\") " Jul 24 23:33:04.944222 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944112 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:33:04.944222 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944157 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home" (OuterVolumeSpecName: "home") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:33:04.944222 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944170 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache" (OuterVolumeSpecName: "model-cache") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:33:04.944365 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944315 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:04.944365 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944331 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:04.944365 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.944341 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:04.945915 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.945891 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm" (OuterVolumeSpecName: "dshm") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:33:04.946022 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.945931 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls" (OuterVolumeSpecName: "kube-api-access-rxqls") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "kube-api-access-rxqls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:33:04.946022 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:04.945970 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8249ea69-7c50-4989-af13-c4195c8e5c4f" (UID: "8249ea69-7c50-4989-af13-c4195c8e5c4f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:33:05.045782 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.045749 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8249ea69-7c50-4989-af13-c4195c8e5c4f-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:05.045782 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.045781 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8249ea69-7c50-4989-af13-c4195c8e5c4f-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:05.045975 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.045791 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxqls\" (UniqueName: \"kubernetes.io/projected/8249ea69-7c50-4989-af13-c4195c8e5c4f-kube-api-access-rxqls\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:05.488308 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.488220 2571 generic.go:356] "Generic (PLEG): container finished" podID="8249ea69-7c50-4989-af13-c4195c8e5c4f" containerID="c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80" exitCode=0 Jul 24 23:33:05.488308 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.488286 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" Jul 24 23:33:05.488663 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.488315 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" event={"ID":"8249ea69-7c50-4989-af13-c4195c8e5c4f","Type":"ContainerDied","Data":"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80"} Jul 24 23:33:05.488663 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.488376 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc" event={"ID":"8249ea69-7c50-4989-af13-c4195c8e5c4f","Type":"ContainerDied","Data":"8212071e4fabc2f9b44c18685d0c3f5a077e3065f1f6fef264555f73a640c3fa"} Jul 24 23:33:05.488663 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.488418 2571 scope.go:117] "RemoveContainer" containerID="c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80" Jul 24 23:33:05.490238 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.490210 2571 generic.go:356] "Generic (PLEG): container finished" podID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" containerID="29bd0a4a49f67a8272238699ca8d2060a955ef96c826f6a6fbd8631cb8ecb0f4" exitCode=0 Jul 24 23:33:05.490352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.490249 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612815312" event={"ID":"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab","Type":"ContainerDied","Data":"29bd0a4a49f67a8272238699ca8d2060a955ef96c826f6a6fbd8631cb8ecb0f4"} Jul 24 23:33:05.493352 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:05.493319 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698616268528\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:05.498029 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.498011 2571 scope.go:117] "RemoveContainer" containerID="c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80" Jul 24 23:33:05.498298 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:05.498266 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80\": container with ID starting with c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80 not found: ID does not exist" containerID="c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80" Jul 24 23:33:05.498358 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.498303 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80"} err="failed to get container status \"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80\": rpc error: code = NotFound desc = could not find container \"c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80\": container with ID starting with c1b5e523a6d9942e559922935638e82be89a8c1ac08384f50f80e719c04efc80 not found: ID does not exist" Jul 24 23:33:05.506216 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:05.506189 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698616268528\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:05.516708 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.516684 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:33:05.522115 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:05.522090 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698616268528\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" pod="opendatahub/test-curl-7613-139698616268528" Jul 24 23:33:05.522545 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.522528 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5c850d/rate-test-model-5c850d-kserve-669989d5c5-zqglc"] Jul 24 23:33:05.562657 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.562627 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea35d58-6be0-42b5-bed4-0a8863c10bed" path="/var/lib/kubelet/pods/4ea35d58-6be0-42b5-bed4-0a8863c10bed/volumes" Jul 24 23:33:05.563040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:05.563025 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8249ea69-7c50-4989-af13-c4195c8e5c4f" path="/var/lib/kubelet/pods/8249ea69-7c50-4989-af13-c4195c8e5c4f/volumes" Jul 24 23:33:06.636028 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.636004 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:06.658855 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.658825 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698612815312_72166c1f-65d8-4b35-a975-e7cf4ff0c7ab/test-curl-7613-139698612815312/0.log" Jul 24 23:33:06.684127 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.684086 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:06.687817 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.687792 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:06.762302 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.762214 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtcxv\" (UniqueName: \"kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv\") pod \"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab\" (UID: \"72166c1f-65d8-4b35-a975-e7cf4ff0c7ab\") " Jul 24 23:33:06.764293 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.764267 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv" (OuterVolumeSpecName: "kube-api-access-qtcxv") pod "72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" (UID: "72166c1f-65d8-4b35-a975-e7cf4ff0c7ab"). InnerVolumeSpecName "kube-api-access-qtcxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:33:06.807241 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807208 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:06.807622 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807609 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8249ea69-7c50-4989-af13-c4195c8e5c4f" containerName="main" Jul 24 23:33:06.807673 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807624 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="8249ea69-7c50-4989-af13-c4195c8e5c4f" containerName="main" Jul 24 23:33:06.807673 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807642 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" containerName="test-curl-7613-139698612815312" Jul 24 23:33:06.807673 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807648 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" containerName="test-curl-7613-139698612815312" Jul 24 23:33:06.807769 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807715 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="8249ea69-7c50-4989-af13-c4195c8e5c4f" containerName="main" Jul 24 23:33:06.807769 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.807723 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" containerName="test-curl-7613-139698612815312" Jul 24 23:33:06.810850 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.810834 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:06.816680 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.816649 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:06.863260 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.863230 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtcxv\" (UniqueName: \"kubernetes.io/projected/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab-kube-api-access-qtcxv\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:06.964050 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:06.964017 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqb8r\" (UniqueName: \"kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r\") pod \"test-curl-7613-139698612815312\" (UID: \"cffc7eb8-9211-403a-9651-af75f120ea1f\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:07.065543 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.065498 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqb8r\" (UniqueName: \"kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r\") pod \"test-curl-7613-139698612815312\" (UID: \"cffc7eb8-9211-403a-9651-af75f120ea1f\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:07.073853 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.073825 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqb8r\" (UniqueName: \"kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r\") pod \"test-curl-7613-139698612815312\" (UID: \"cffc7eb8-9211-403a-9651-af75f120ea1f\") " pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:07.121540 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.121507 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:07.252192 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.252166 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:07.254371 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:33:07.254336 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcffc7eb8_9211_403a_9651_af75f120ea1f.slice/crio-76bbfa5e29c64f8a93de937981ab7f53755f389113ff1526c6ffff8807609aa9 WatchSource:0}: Error finding container 76bbfa5e29c64f8a93de937981ab7f53755f389113ff1526c6ffff8807609aa9: Status 404 returned error can't find the container with id 76bbfa5e29c64f8a93de937981ab7f53755f389113ff1526c6ffff8807609aa9 Jul 24 23:33:07.502766 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.502662 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:07.502766 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.502671 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c56a0d5016065112d81017cfa1d5f125a8c26985d95f337224fde07256e4c43c" Jul 24 23:33:07.504024 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.503993 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612815312" event={"ID":"cffc7eb8-9211-403a-9651-af75f120ea1f","Type":"ContainerStarted","Data":"76bbfa5e29c64f8a93de937981ab7f53755f389113ff1526c6ffff8807609aa9"} Jul 24 23:33:07.514421 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.514381 2571 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7613-139698612815312" oldPodUID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" Jul 24 23:33:07.560180 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:07.560150 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72166c1f-65d8-4b35-a975-e7cf4ff0c7ab" path="/var/lib/kubelet/pods/72166c1f-65d8-4b35-a975-e7cf4ff0c7ab/volumes" Jul 24 23:33:08.511305 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:08.511267 2571 generic.go:356] "Generic (PLEG): container finished" podID="cffc7eb8-9211-403a-9651-af75f120ea1f" containerID="04fba9d344add429041046687b6d01f707d0396abf0a0dff2118999cfc7f053b" exitCode=0 Jul 24 23:33:08.511725 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:08.511346 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7613-139698612815312" event={"ID":"cffc7eb8-9211-403a-9651-af75f120ea1f","Type":"ContainerDied","Data":"04fba9d344add429041046687b6d01f707d0396abf0a0dff2118999cfc7f053b"} Jul 24 23:33:09.655783 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.655753 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:09.676070 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.676040 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7613-139698612815312_cffc7eb8-9211-403a-9651-af75f120ea1f/test-curl-7613-139698612815312/0.log" Jul 24 23:33:09.700744 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.700711 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:09.702584 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.702564 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7613-139698612815312"] Jul 24 23:33:09.793966 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.793933 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqb8r\" (UniqueName: \"kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r\") pod \"cffc7eb8-9211-403a-9651-af75f120ea1f\" (UID: \"cffc7eb8-9211-403a-9651-af75f120ea1f\") " Jul 24 23:33:09.796041 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.796008 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r" (OuterVolumeSpecName: "kube-api-access-pqb8r") pod "cffc7eb8-9211-403a-9651-af75f120ea1f" (UID: "cffc7eb8-9211-403a-9651-af75f120ea1f"). InnerVolumeSpecName "kube-api-access-pqb8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:33:09.895009 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:09.894978 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqb8r\" (UniqueName: \"kubernetes.io/projected/cffc7eb8-9211-403a-9651-af75f120ea1f-kube-api-access-pqb8r\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:33:10.522200 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:10.522173 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76bbfa5e29c64f8a93de937981ab7f53755f389113ff1526c6ffff8807609aa9" Jul 24 23:33:10.522200 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:10.522192 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:10.531797 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:10.531769 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698612815312\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:11.557874 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:33:11.557837 2571 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7613-139698612815312\" is forbidden: User \"system:node:ip-10-0-136-126.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-126.ec2.internal' and this object" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" pod="opendatahub/test-curl-7613-139698612815312" Jul 24 23:33:11.559850 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:11.559826 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" path="/var/lib/kubelet/pods/cffc7eb8-9211-403a-9651-af75f120ea1f/volumes" Jul 24 23:33:12.637975 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:12.637945 2571 scope.go:117] "RemoveContainer" containerID="4165c71cec61b2fe7f98b79f0280769e71d9e86c00551f47dbd74b9b63204732" Jul 24 23:33:28.675025 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.674988 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:33:28.675527 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.675453 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" containerName="test-curl-7613-139698612815312" Jul 24 23:33:28.675527 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.675467 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" containerName="test-curl-7613-139698612815312" Jul 24 23:33:28.675615 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.675539 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffc7eb8-9211-403a-9651-af75f120ea1f" containerName="test-curl-7613-139698612815312" Jul 24 23:33:28.680466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.680445 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.682486 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.682466 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-064932-openshift-default-dockercfg-f6dq4\"" Jul 24 23:33:28.682597 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.682511 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 23:33:28.691541 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.691516 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:33:28.768457 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768382 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768457 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768454 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768474 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8srj\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768512 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768556 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768598 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768644 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768685 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768675 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.768898 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.768694 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.869824 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.869787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.869850 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.869880 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.869897 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.869911 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870198 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870028 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870198 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870066 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870198 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870090 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8srj\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870198 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870132 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870432 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870237 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870432 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870282 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870432 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870347 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870595 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870501 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.870686 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.870669 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.872375 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.872355 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.872528 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.872506 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.878185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.878164 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.878363 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.878345 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8srj\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj\") pod \"e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:28.991578 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:28.991498 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:29.144297 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:29.144274 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:33:29.146251 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:33:29.146227 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bab1b4_f417_40f9_bf0e_fda91efd142f.slice/crio-b4f6b5d433d6e05fca80e349b21efbe1f77b1025e0e0caa8beb05b1605a2aac2 WatchSource:0}: Error finding container b4f6b5d433d6e05fca80e349b21efbe1f77b1025e0e0caa8beb05b1605a2aac2: Status 404 returned error can't find the container with id b4f6b5d433d6e05fca80e349b21efbe1f77b1025e0e0caa8beb05b1605a2aac2 Jul 24 23:33:29.618348 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:29.618312 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerStarted","Data":"b4f6b5d433d6e05fca80e349b21efbe1f77b1025e0e0caa8beb05b1605a2aac2"} Jul 24 23:33:47.382884 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.382840 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:33:47.383225 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.382921 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:33:47.383225 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.382948 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:33:47.714032 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.713945 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerStarted","Data":"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1"} Jul 24 23:33:47.735011 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.734963 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podStartSLOduration=1.5008108610000002 podStartE2EDuration="19.734949397s" totalImagesPullingTime="18.234138536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:47.732939134 +0000 UTC m=+2856.662278346" watchObservedRunningTime="2026-07-24 23:33:47.734949397 +0000 UTC m=+2856.664288610" Jul 24 23:33:47.992635 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:47.992542 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:48.997235 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:48.997202 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:49.724455 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:49.724419 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:49.725454 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:49.725436 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:33:50.595932 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.595894 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:33:50.599726 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.599710 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.601642 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.601619 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-064932\"" Jul 24 23:33:50.601748 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.601644 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-064932-dockercfg-p6sl4\"" Jul 24 23:33:50.610176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.610154 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:33:50.696270 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.696237 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.696459 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.696373 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.797609 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.797570 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.797825 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.797633 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.798287 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.798261 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.805119 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.805086 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg\") pod \"payload-processing-e2e-ipp-b-064932-77d5569676-whvjk\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:50.910964 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:50.910868 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:51.049112 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:51.049085 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:33:51.051888 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:33:51.051857 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1eb5d0_2abe_4f27_ac0c_a696a70b46a1.slice/crio-1667429bf05010ae8c6259729156a2d1e7d4000fb8ccef6a355d3431dee4e830 WatchSource:0}: Error finding container 1667429bf05010ae8c6259729156a2d1e7d4000fb8ccef6a355d3431dee4e830: Status 404 returned error can't find the container with id 1667429bf05010ae8c6259729156a2d1e7d4000fb8ccef6a355d3431dee4e830 Jul 24 23:33:51.737818 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:51.737759 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" event={"ID":"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1","Type":"ContainerStarted","Data":"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35"} Jul 24 23:33:51.738316 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:51.737834 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" event={"ID":"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1","Type":"ContainerStarted","Data":"1667429bf05010ae8c6259729156a2d1e7d4000fb8ccef6a355d3431dee4e830"} Jul 24 23:33:51.738316 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:51.737965 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:33:51.758575 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:33:51.758517 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" podStartSLOduration=1.7585007670000001 podStartE2EDuration="1.758500767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:51.756257071 +0000 UTC m=+2860.685596303" watchObservedRunningTime="2026-07-24 23:33:51.758500767 +0000 UTC m=+2860.687839979" Jul 24 23:34:02.744919 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.744892 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:34:02.796004 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.795969 2571 generic.go:356] "Generic (PLEG): container finished" podID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerID="756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1" exitCode=0 Jul 24 23:34:02.796163 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.796038 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerDied","Data":"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1"} Jul 24 23:34:02.796474 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.796457 2571 scope.go:117] "RemoveContainer" containerID="756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1" Jul 24 23:34:02.799200 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.799167 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:34:02.799297 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.799240 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:34:02.799297 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:02.799273 2571 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 24 23:34:03.803009 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:03.802975 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerStarted","Data":"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310"} Jul 24 23:34:03.991956 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:03.991922 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:34:03.993200 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:03.993165 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 23:34:03.993352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:03.993230 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 23:34:04.725492 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:04.725407 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:34:04.992762 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:04.992673 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 23:34:04.992762 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:04.992732 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 23:34:05.992819 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:05.992778 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 23:34:05.993298 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:05.992861 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 23:34:06.992046 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:06.992004 2571 patch_prober.go:28] interesting pod/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 23:34:06.992256 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:06.992081 2571 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 23:34:07.996672 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:07.996640 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:34:07.997600 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:07.997579 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:34:11.285706 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:11.285673 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:34:11.401473 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:11.401441 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-064932-77d5569676-whvjk_2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1/payload-processing/0.log" Jul 24 23:34:22.755080 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.755040 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:34:22.759204 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.759184 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.761166 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.761142 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eb4c8d\"/\"openshift-service-ca.crt\"" Jul 24 23:34:22.761304 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.761145 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-eb4c8d\"/\"kube-root-ca.crt\"" Jul 24 23:34:22.761965 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.761947 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eb4c8d\"/\"ipp-route-eb4c8d-kserve-self-signed-certs\"" Jul 24 23:34:22.762149 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.761951 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-eb4c8d\"/\"default-dockercfg-jsqkm\"" Jul 24 23:34:22.765085 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.765060 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:34:22.940863 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.940827 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5xjz\" (UniqueName: \"kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.940863 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.940865 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.941081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.940936 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.941081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.940966 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.941081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.941008 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:22.941081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:22.941052 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042378 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042344 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5xjz\" (UniqueName: \"kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042563 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042385 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042563 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042444 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042563 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042467 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042563 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042492 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042709 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042553 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042860 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042839 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.042988 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042969 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.043040 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.042970 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.044759 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.044739 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.044945 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.044926 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.049892 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.049870 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5xjz\" (UniqueName: \"kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz\") pod \"ipp-route-eb4c8d-kserve-749bd95955-tlgmr\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.070036 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.070013 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:23.203562 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.203534 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:34:23.205495 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:34:23.205463 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2daba9df_082a_4dd9_ae17_e4fc53b11304.slice/crio-4759b779409baa8f76cec0ac4bcc0b0f781f30c5495d40c62e5ebe46bca9a6d6 WatchSource:0}: Error finding container 4759b779409baa8f76cec0ac4bcc0b0f781f30c5495d40c62e5ebe46bca9a6d6: Status 404 returned error can't find the container with id 4759b779409baa8f76cec0ac4bcc0b0f781f30c5495d40c62e5ebe46bca9a6d6 Jul 24 23:34:23.912837 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.912802 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" event={"ID":"2daba9df-082a-4dd9-ae17-e4fc53b11304","Type":"ContainerStarted","Data":"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8"} Jul 24 23:34:23.913234 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.912845 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" event={"ID":"2daba9df-082a-4dd9-ae17-e4fc53b11304","Type":"ContainerStarted","Data":"4759b779409baa8f76cec0ac4bcc0b0f781f30c5495d40c62e5ebe46bca9a6d6"} Jul 24 23:34:23.931126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:23.931076 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" podStartSLOduration=1.931061083 podStartE2EDuration="1.931061083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:23.929356623 +0000 UTC m=+2892.858695835" watchObservedRunningTime="2026-07-24 23:34:23.931061083 +0000 UTC m=+2892.860400295" Jul 24 23:34:33.070674 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:33.070633 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:33.071237 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:33.070720 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:33.082966 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:33.082944 2571 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:33.981074 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:33.981046 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:34:53.162252 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:53.162212 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:34:53.278032 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:34:53.278003 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-064932-77d5569676-whvjk_2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1/payload-processing/0.log" Jul 24 23:35:11.976047 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:11.976009 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr"] Jul 24 23:35:11.979938 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:11.979921 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:11.981732 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:11.981710 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\"" Jul 24 23:35:11.985095 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:11.985067 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr"] Jul 24 23:35:12.031724 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:12.031690 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr\" (UID: \"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:12.133272 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:12.133234 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr\" (UID: \"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:12.133899 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:12.133879 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr\" (UID: \"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:12.291727 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:12.291691 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:12.435914 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:12.435679 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr"] Jul 24 23:35:12.438873 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:35:12.438842 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ba3fd6_91d4_4f7d_829d_ac2738d966fd.slice/crio-1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293 WatchSource:0}: Error finding container 1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293: Status 404 returned error can't find the container with id 1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293 Jul 24 23:35:13.171355 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:13.171320 2571 generic.go:356] "Generic (PLEG): container finished" podID="c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" containerID="073bade61e93d259996bf98280132b3627c54417efd401cb76eda2d12e7ee82b" exitCode=0 Jul 24 23:35:13.171721 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:13.171414 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" event={"ID":"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd","Type":"ContainerDied","Data":"073bade61e93d259996bf98280132b3627c54417efd401cb76eda2d12e7ee82b"} Jul 24 23:35:13.171721 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:13.171471 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" event={"ID":"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd","Type":"ContainerStarted","Data":"1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293"} Jul 24 23:35:14.313002 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:14.312967 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:14.358441 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:14.358404 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca\") pod \"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd\" (UID: \"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd\") " Jul 24 23:35:14.358760 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:14.358731 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" (UID: "c3ba3fd6-91d4-4f7d-829d-ac2738d966fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:35:14.358841 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:14.358825 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:35:15.182485 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:15.182448 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" event={"ID":"c3ba3fd6-91d4-4f7d-829d-ac2738d966fd","Type":"ContainerDied","Data":"1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293"} Jul 24 23:35:15.182662 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:15.182506 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bbddb64ad0bb5f2cf0c159fd0dd8be5e6866b3f72ac723e69678c1e939bc293" Jul 24 23:35:15.182662 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:15.182466 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr" Jul 24 23:35:27.051885 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:27.051842 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr"] Jul 24 23:35:27.054120 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:27.054094 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-429acb-jlzxr"] Jul 24 23:35:27.560737 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:35:27.560705 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" path="/var/lib/kubelet/pods/c3ba3fd6-91d4-4f7d-829d-ac2738d966fd/volumes" Jul 24 23:36:02.433898 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:02.433825 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:36:12.601190 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:12.601161 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:36:12.606514 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:12.606487 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:36:12.609680 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:12.609662 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:36:12.616351 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:12.616332 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-85sr5_e411847a-4117-4ccb-8b67-93fb9eca114d/ovn-acl-logging/0.log" Jul 24 23:36:17.453003 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.452956 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" podUID="2daba9df-082a-4dd9-ae17-e4fc53b11304" containerName="main" containerID="cri-o://75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8" gracePeriod=45 Jul 24 23:36:17.710267 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.710202 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:36:17.800333 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800299 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800348 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800375 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800408 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800461 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800526 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800499 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r5xjz\" (UniqueName: \"kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz\") pod \"2daba9df-082a-4dd9-ae17-e4fc53b11304\" (UID: \"2daba9df-082a-4dd9-ae17-e4fc53b11304\") " Jul 24 23:36:17.800778 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800600 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache" (OuterVolumeSpecName: "model-cache") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:17.800778 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800651 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:17.800778 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800679 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home" (OuterVolumeSpecName: "home") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:17.800886 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800877 2571 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-model-cache\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:17.800920 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800889 2571 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-home\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:17.800920 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.800901 2571 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-tmp-dir\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:17.802596 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.802566 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm" (OuterVolumeSpecName: "dshm") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:17.802713 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.802592 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz" (OuterVolumeSpecName: "kube-api-access-r5xjz") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "kube-api-access-r5xjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:36:17.802713 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.802658 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2daba9df-082a-4dd9-ae17-e4fc53b11304" (UID: "2daba9df-082a-4dd9-ae17-e4fc53b11304"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:36:17.901935 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.901886 2571 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2daba9df-082a-4dd9-ae17-e4fc53b11304-tls-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:17.901935 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.901925 2571 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2daba9df-082a-4dd9-ae17-e4fc53b11304-dshm\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:17.901935 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:17.901936 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5xjz\" (UniqueName: \"kubernetes.io/projected/2daba9df-082a-4dd9-ae17-e4fc53b11304-kube-api-access-r5xjz\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:36:18.521596 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.521557 2571 generic.go:356] "Generic (PLEG): container finished" podID="2daba9df-082a-4dd9-ae17-e4fc53b11304" containerID="75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8" exitCode=0 Jul 24 23:36:18.521596 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.521601 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" Jul 24 23:36:18.522146 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.521642 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" event={"ID":"2daba9df-082a-4dd9-ae17-e4fc53b11304","Type":"ContainerDied","Data":"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8"} Jul 24 23:36:18.522146 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.521699 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr" event={"ID":"2daba9df-082a-4dd9-ae17-e4fc53b11304","Type":"ContainerDied","Data":"4759b779409baa8f76cec0ac4bcc0b0f781f30c5495d40c62e5ebe46bca9a6d6"} Jul 24 23:36:18.522146 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.521726 2571 scope.go:117] "RemoveContainer" containerID="75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8" Jul 24 23:36:18.531473 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.531450 2571 scope.go:117] "RemoveContainer" containerID="75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8" Jul 24 23:36:18.531741 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:36:18.531722 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8\": container with ID starting with 75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8 not found: ID does not exist" containerID="75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8" Jul 24 23:36:18.531783 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.531751 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8"} err="failed to get container status \"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8\": rpc error: code = NotFound desc = could not find container \"75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8\": container with ID starting with 75265010ab9200e27da35b643c3924d414455ef526c9927955d88ee4035265b8 not found: ID does not exist" Jul 24 23:36:18.541417 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.541368 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:36:18.545134 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:18.545109 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-eb4c8d/ipp-route-eb4c8d-kserve-749bd95955-tlgmr"] Jul 24 23:36:19.561294 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:36:19.561252 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2daba9df-082a-4dd9-ae17-e4fc53b11304" path="/var/lib/kubelet/pods/2daba9df-082a-4dd9-ae17-e4fc53b11304/volumes" Jul 24 23:37:01.394665 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.394625 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:37:01.395286 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.394924 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" podUID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" containerName="payload-processing" containerID="cri-o://89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35" gracePeriod=30 Jul 24 23:37:01.657374 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.657297 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:37:01.725627 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.725595 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume\") pod \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " Jul 24 23:37:01.725820 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.725752 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg\") pod \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\" (UID: \"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1\") " Jul 24 23:37:01.725953 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.725925 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" (UID: "2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:37:01.726130 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.726117 2571 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-plugins-config-volume\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:01.728050 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.728018 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg" (OuterVolumeSpecName: "kube-api-access-zwjhg") pod "2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" (UID: "2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1"). InnerVolumeSpecName "kube-api-access-zwjhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:37:01.751467 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.751431 2571 generic.go:356] "Generic (PLEG): container finished" podID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" containerID="89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35" exitCode=0 Jul 24 23:37:01.751623 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.751477 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" event={"ID":"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1","Type":"ContainerDied","Data":"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35"} Jul 24 23:37:01.751623 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.751514 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" event={"ID":"2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1","Type":"ContainerDied","Data":"1667429bf05010ae8c6259729156a2d1e7d4000fb8ccef6a355d3431dee4e830"} Jul 24 23:37:01.751623 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.751526 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk" Jul 24 23:37:01.751777 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.751534 2571 scope.go:117] "RemoveContainer" containerID="89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35" Jul 24 23:37:01.767601 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.767536 2571 scope.go:117] "RemoveContainer" containerID="89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35" Jul 24 23:37:01.768129 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:37:01.768093 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35\": container with ID starting with 89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35 not found: ID does not exist" containerID="89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35" Jul 24 23:37:01.768251 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.768139 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35"} err="failed to get container status \"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35\": rpc error: code = NotFound desc = could not find container \"89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35\": container with ID starting with 89cb28732677706b19833b5e1f7fbf864be646e9d3c8ef10d8a81ed1355a0e35 not found: ID does not exist" Jul 24 23:37:01.778651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.778622 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:37:01.782973 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.782944 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-064932-77d5569676-whvjk"] Jul 24 23:37:01.826586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:01.826552 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwjhg\" (UniqueName: \"kubernetes.io/projected/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1-kube-api-access-zwjhg\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:03.560287 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:03.560241 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" path="/var/lib/kubelet/pods/2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1/volumes" Jul 24 23:37:12.479314 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.479276 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:37:12.479852 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.479594 2571 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" containerID="cri-o://9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310" gracePeriod=30 Jul 24 23:37:12.861176 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861139 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt"] Jul 24 23:37:12.861598 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861583 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" containerName="revoke-keys" Jul 24 23:37:12.861651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861600 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" containerName="revoke-keys" Jul 24 23:37:12.861651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861631 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" containerName="payload-processing" Jul 24 23:37:12.861651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861637 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" containerName="payload-processing" Jul 24 23:37:12.861651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861645 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2daba9df-082a-4dd9-ae17-e4fc53b11304" containerName="main" Jul 24 23:37:12.861651 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861650 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="2daba9df-082a-4dd9-ae17-e4fc53b11304" containerName="main" Jul 24 23:37:12.861800 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861707 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c1eb5d0-2abe-4f27-ac0c-a696a70b46a1" containerName="payload-processing" Jul 24 23:37:12.861800 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861714 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3ba3fd6-91d4-4f7d-829d-ac2738d966fd" containerName="revoke-keys" Jul 24 23:37:12.861800 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.861723 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="2daba9df-082a-4dd9-ae17-e4fc53b11304" containerName="main" Jul 24 23:37:12.871524 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.871492 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:12.874064 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.874030 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt"] Jul 24 23:37:12.875010 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.874982 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vbbt6\"" Jul 24 23:37:12.937729 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:12.937688 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt\" (UID: \"3a32da7b-7ba0-49c9-a8ec-fb25092904d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:13.038827 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.038787 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt\" (UID: \"3a32da7b-7ba0-49c9-a8ec-fb25092904d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:13.039490 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.039473 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt\" (UID: \"3a32da7b-7ba0-49c9-a8ec-fb25092904d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:13.185367 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.185279 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:13.324368 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.324229 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt"] Jul 24 23:37:13.326836 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:37:13.326788 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a32da7b_7ba0_49c9_a8ec_fb25092904d2.slice/crio-8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99 WatchSource:0}: Error finding container 8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99: Status 404 returned error can't find the container with id 8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99 Jul 24 23:37:13.818233 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.818144 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" event={"ID":"3a32da7b-7ba0-49c9-a8ec-fb25092904d2","Type":"ContainerStarted","Data":"4c04ef9d9360444454a880d4931a2393f6e61550eb1781ae2dcaa955aeb4e33d"} Jul 24 23:37:13.818233 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.818211 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" event={"ID":"3a32da7b-7ba0-49c9-a8ec-fb25092904d2","Type":"ContainerStarted","Data":"8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99"} Jul 24 23:37:13.831835 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:13.831776 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" podStartSLOduration=1.8317556179999999 podStartE2EDuration="1.831755618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:37:13.830956355 +0000 UTC m=+3062.760295566" watchObservedRunningTime="2026-07-24 23:37:13.831755618 +0000 UTC m=+3062.761094833" Jul 24 23:37:14.827439 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:14.827386 2571 generic.go:356] "Generic (PLEG): container finished" podID="3a32da7b-7ba0-49c9-a8ec-fb25092904d2" containerID="4c04ef9d9360444454a880d4931a2393f6e61550eb1781ae2dcaa955aeb4e33d" exitCode=0 Jul 24 23:37:14.827843 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:14.827485 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" event={"ID":"3a32da7b-7ba0-49c9-a8ec-fb25092904d2","Type":"ContainerDied","Data":"4c04ef9d9360444454a880d4931a2393f6e61550eb1781ae2dcaa955aeb4e33d"} Jul 24 23:37:15.970862 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:15.970829 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:16.070234 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.070202 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca\") pod \"3a32da7b-7ba0-49c9-a8ec-fb25092904d2\" (UID: \"3a32da7b-7ba0-49c9-a8ec-fb25092904d2\") " Jul 24 23:37:16.070582 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.070557 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a32da7b-7ba0-49c9-a8ec-fb25092904d2" (UID: "3a32da7b-7ba0-49c9-a8ec-fb25092904d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:37:16.171438 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.171312 2571 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a32da7b-7ba0-49c9-a8ec-fb25092904d2-maas-api-service-ca\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:16.839223 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.839189 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" Jul 24 23:37:16.839223 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.839197 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt" event={"ID":"3a32da7b-7ba0-49c9-a8ec-fb25092904d2","Type":"ContainerDied","Data":"8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99"} Jul 24 23:37:16.839479 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:16.839243 2571 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba9f4a77a42f0c3ee30d525c37e7ca7231d0f2a37ba13318a94c414c97a7e99" Jul 24 23:37:17.741185 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.741159 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:37:17.852352 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.852315 2571 generic.go:356] "Generic (PLEG): container finished" podID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerID="9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310" exitCode=0 Jul 24 23:37:17.852601 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.852368 2571 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" Jul 24 23:37:17.852601 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.852414 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerDied","Data":"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310"} Jul 24 23:37:17.852601 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.852488 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx" event={"ID":"36bab1b4-f417-40f9-bf0e-fda91efd142f","Type":"ContainerDied","Data":"b4f6b5d433d6e05fca80e349b21efbe1f77b1025e0e0caa8beb05b1605a2aac2"} Jul 24 23:37:17.852601 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.852511 2571 scope.go:117] "RemoveContainer" containerID="9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310" Jul 24 23:37:17.861458 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.861438 2571 scope.go:117] "RemoveContainer" containerID="756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1" Jul 24 23:37:17.869555 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.869537 2571 scope.go:117] "RemoveContainer" containerID="9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310" Jul 24 23:37:17.869804 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:37:17.869785 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310\": container with ID starting with 9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310 not found: ID does not exist" containerID="9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310" Jul 24 23:37:17.869849 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.869812 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310"} err="failed to get container status \"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310\": rpc error: code = NotFound desc = could not find container \"9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310\": container with ID starting with 9d6b3b38f2567dd25c25f50c64fb7ddfc44c130336e25a31fb3a953500b71310 not found: ID does not exist" Jul 24 23:37:17.869849 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.869829 2571 scope.go:117] "RemoveContainer" containerID="756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1" Jul 24 23:37:17.870046 ip-10-0-136-126 kubenswrapper[2571]: E0724 23:37:17.870030 2571 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1\": container with ID starting with 756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1 not found: ID does not exist" containerID="756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1" Jul 24 23:37:17.870092 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.870047 2571 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1"} err="failed to get container status \"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1\": rpc error: code = NotFound desc = could not find container \"756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1\": container with ID starting with 756b0db55197b5bc2140c27f285be48dd5b76405e04d81f60242fb722b7be6b1 not found: ID does not exist" Jul 24 23:37:17.888404 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888362 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8srj\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888544 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888436 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888544 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888466 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888544 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888481 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888544 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888506 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888758 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888552 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888758 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888575 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888758 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888598 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888758 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888629 2571 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket\") pod \"36bab1b4-f417-40f9-bf0e-fda91efd142f\" (UID: \"36bab1b4-f417-40f9-bf0e-fda91efd142f\") " Jul 24 23:37:17.888958 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.888928 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:37:17.889092 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.889034 2571 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36bab1b4-f417-40f9-bf0e-fda91efd142f-istiod-ca-cert\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.889178 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.889154 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:37:17.889302 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.889282 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:37:17.889434 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.889413 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data" (OuterVolumeSpecName: "istio-data") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:37:17.889507 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.889440 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:37:17.890758 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.890734 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj" (OuterVolumeSpecName: "kube-api-access-h8srj") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "kube-api-access-h8srj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:37:17.890871 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.890773 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:37:17.890984 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.890965 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:37:17.891027 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.890982 2571 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token" (OuterVolumeSpecName: "istio-token") pod "36bab1b4-f417-40f9-bf0e-fda91efd142f" (UID: "36bab1b4-f417-40f9-bf0e-fda91efd142f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:37:17.990374 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990326 2571 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8srj\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-kube-api-access-h8srj\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990374 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990368 2571 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-podinfo\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990374 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990380 2571 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-envoy\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990417 2571 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-credential-socket\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990431 2571 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-certs\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990442 2571 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-token\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990452 2571 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-istio-data\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.990694 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:17.990462 2571 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36bab1b4-f417-40f9-bf0e-fda91efd142f-workload-socket\") on node \"ip-10-0-136-126.ec2.internal\" DevicePath \"\"" Jul 24 23:37:18.174121 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:18.174085 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:37:18.177383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:18.177356 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-064932-openshift-default-5db6b488b4-xtnwx"] Jul 24 23:37:19.559876 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:19.559830 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" path="/var/lib/kubelet/pods/36bab1b4-f417-40f9-bf0e-fda91efd142f/volumes" Jul 24 23:37:27.939184 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:27.939127 2571 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt"] Jul 24 23:37:27.941828 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:27.941792 2571 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5c850d-w5wpt"] Jul 24 23:37:29.560100 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:37:29.560067 2571 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a32da7b-7ba0-49c9-a8ec-fb25092904d2" path="/var/lib/kubelet/pods/3a32da7b-7ba0-49c9-a8ec-fb25092904d2/volumes" Jul 24 23:38:02.144996 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:02.144961 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9dbb4674-s44gl_524fb69a-b2e2-4838-8e37-c4fbe1270a5f/manager/0.log" Jul 24 23:38:02.615175 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:02.615146 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-9fbmj_1066e233-706a-4088-a4fe-fbd5a7aac431/copy-manifests/0.log" Jul 24 23:38:02.621979 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:02.621947 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-9fbmj_1066e233-706a-4088-a4fe-fbd5a7aac431/manager/0.log" Jul 24 23:38:03.094370 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:03.094335 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d4c8b85cc-gb9wh_903ff18a-64a5-4f10-99c7-1dbfb0961057/maas-api/0.log" Jul 24 23:38:03.538107 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:03.538073 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-lhlg2_1ae7c29b-fa23-43fc-ae2a-cf3edf8499fc/postgres/0.log" Jul 24 23:38:05.148202 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:05.148176 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mjgkg_b4e83a82-4e36-4721-9f09-de0099368511/kuadrant-console-plugin/0.log" Jul 24 23:38:06.209213 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:06.209174 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-f7q7t_9cff92f8-a35f-411a-8cca-4155434c3ed9/kube-auth-proxy/0.log" Jul 24 23:38:06.561750 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:06.561718 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cfnx4_47034fdc-50f0-45b9-9ca7-354ebce2ec89/payload-pre-processing/0.log" Jul 24 23:38:06.689198 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:06.689166 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:38:06.810152 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:06.810122 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-88d7b5c95-jqkth_5eaa6fba-900c-4fb8-b51c-821f7112fd03/router/0.log" Jul 24 23:38:07.272583 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:07.272555 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vp2n9_b98fc85f-290e-42a0-88c9-70ec55212176/main/0.log" Jul 24 23:38:07.504409 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:07.504368 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fzx9t7_70bfc930-2e9c-46ea-936d-26f47ccacd14/main/0.log" Jul 24 23:38:07.735440 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:07.735333 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cw25j_cbd7d196-5a8b-45c6-8306-a72e216f1904/main/0.log" Jul 24 23:38:16.197893 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:16.197854 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-tvlk8_1aa6fd6f-f248-4fb2-bffd-3c3c83345e0c/global-pull-secret-syncer/0.log" Jul 24 23:38:16.319737 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:16.319696 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hvwbp_fb05b4bb-3520-468d-b4c6-2290d6293cf5/konnectivity-agent/0.log" Jul 24 23:38:16.370751 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:16.370720 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-126.ec2.internal_bf90ceb74d2562be9fba76df0fbf255c/haproxy/0.log" Jul 24 23:38:22.156884 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:22.156852 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-mjgkg_b4e83a82-4e36-4721-9f09-de0099368511/kuadrant-console-plugin/0.log" Jul 24 23:38:23.998581 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:23.998498 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/alertmanager/0.log" Jul 24 23:38:24.022918 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.022868 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/config-reloader/0.log" Jul 24 23:38:24.045117 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.045085 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/kube-rbac-proxy-web/0.log" Jul 24 23:38:24.067330 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.067307 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/kube-rbac-proxy/0.log" Jul 24 23:38:24.095384 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.095360 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/kube-rbac-proxy-metric/0.log" Jul 24 23:38:24.117331 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.117307 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/prom-label-proxy/0.log" Jul 24 23:38:24.138076 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.138049 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_8fa88a77-56e7-4e6b-826c-1c33a6ea31c7/init-config-reloader/0.log" Jul 24 23:38:24.188021 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.187994 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-slws5_5092f984-2bce-4b55-a5c9-f3e69b726b07/cluster-monitoring-operator/0.log" Jul 24 23:38:24.282580 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.282550 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f58db8566-zmzls_e16a8a9d-bea3-4ed7-a148-c32af05fdf0d/metrics-server/0.log" Jul 24 23:38:24.475631 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.475600 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzlwv_46bc6819-d493-49bd-b65b-449051088bd2/node-exporter/0.log" Jul 24 23:38:24.497800 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.497773 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzlwv_46bc6819-d493-49bd-b65b-449051088bd2/kube-rbac-proxy/0.log" Jul 24 23:38:24.518857 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.518830 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-wzlwv_46bc6819-d493-49bd-b65b-449051088bd2/init-textfile/0.log" Jul 24 23:38:24.543172 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.543101 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-scmkd_e3d8b8e1-2499-4bd1-ad2a-68146dbb599f/kube-rbac-proxy-main/0.log" Jul 24 23:38:24.563382 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.563352 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-scmkd_e3d8b8e1-2499-4bd1-ad2a-68146dbb599f/kube-rbac-proxy-self/0.log" Jul 24 23:38:24.584216 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.584185 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-scmkd_e3d8b8e1-2499-4bd1-ad2a-68146dbb599f/openshift-state-metrics/0.log" Jul 24 23:38:24.621031 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.621004 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/prometheus/0.log" Jul 24 23:38:24.644410 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.644377 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/config-reloader/0.log" Jul 24 23:38:24.665529 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.665501 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/thanos-sidecar/0.log" Jul 24 23:38:24.685533 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.685478 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/kube-rbac-proxy-web/0.log" Jul 24 23:38:24.707709 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.707683 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/kube-rbac-proxy/0.log" Jul 24 23:38:24.729120 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.729092 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/kube-rbac-proxy-thanos/0.log" Jul 24 23:38:24.753286 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:24.753232 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0a57d006-50c6-476d-a383-3e0a6081db7d/init-config-reloader/0.log" Jul 24 23:38:26.007985 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.007948 2571 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg"] Jul 24 23:38:26.008362 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008351 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" Jul 24 23:38:26.008362 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008362 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" Jul 24 23:38:26.008467 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008384 2571 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a32da7b-7ba0-49c9-a8ec-fb25092904d2" containerName="revoke-keys" Jul 24 23:38:26.008467 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008406 2571 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a32da7b-7ba0-49c9-a8ec-fb25092904d2" containerName="revoke-keys" Jul 24 23:38:26.008467 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008461 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="36bab1b4-f417-40f9-bf0e-fda91efd142f" containerName="istio-proxy" Jul 24 23:38:26.008556 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.008470 2571 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a32da7b-7ba0-49c9-a8ec-fb25092904d2" containerName="revoke-keys" Jul 24 23:38:26.011652 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.011629 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.013862 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.013843 2571 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qntww\"/\"default-dockercfg-cs6ws\"" Jul 24 23:38:26.013991 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.013887 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qntww\"/\"kube-root-ca.crt\"" Jul 24 23:38:26.013991 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.013929 2571 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qntww\"/\"openshift-service-ca.crt\"" Jul 24 23:38:26.018097 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.018064 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg"] Jul 24 23:38:26.153604 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.153570 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-lib-modules\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.153794 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.153608 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-podres\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.153794 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.153663 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-proc\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.153794 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.153785 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5znj\" (UniqueName: \"kubernetes.io/projected/39dead86-d402-4c70-9615-4525ba1588fd-kube-api-access-m5znj\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.153932 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.153818 2571 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-sys\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.206984 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.206952 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-bx5rd_2d08fc13-992d-4c02-b9a1-a06ac9bec210/networking-console-plugin/0.log" Jul 24 23:38:26.255308 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255265 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-sys\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255504 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255359 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-sys\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255504 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255375 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-lib-modules\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255504 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255422 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-podres\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255504 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255477 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-proc\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255504 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255494 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-lib-modules\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255720 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255548 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-proc\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255720 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255577 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39dead86-d402-4c70-9615-4525ba1588fd-podres\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.255720 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.255593 2571 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5znj\" (UniqueName: \"kubernetes.io/projected/39dead86-d402-4c70-9615-4525ba1588fd-kube-api-access-m5znj\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.262317 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.262249 2571 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5znj\" (UniqueName: \"kubernetes.io/projected/39dead86-d402-4c70-9615-4525ba1588fd-kube-api-access-m5znj\") pod \"perf-node-gather-daemonset-j76dg\" (UID: \"39dead86-d402-4c70-9615-4525ba1588fd\") " pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.322477 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.322446 2571 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:26.447815 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.447783 2571 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg"] Jul 24 23:38:26.451010 ip-10-0-136-126 kubenswrapper[2571]: W0724 23:38:26.450986 2571 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod39dead86_d402_4c70_9615_4525ba1588fd.slice/crio-1fd10153fdd7743cc3112d03d852ce344e326cf92ab4d96d6b91078139384bf8 WatchSource:0}: Error finding container 1fd10153fdd7743cc3112d03d852ce344e326cf92ab4d96d6b91078139384bf8: Status 404 returned error can't find the container with id 1fd10153fdd7743cc3112d03d852ce344e326cf92ab4d96d6b91078139384bf8 Jul 24 23:38:26.810353 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.810257 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/2.log" Jul 24 23:38:26.818641 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:26.818616 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-ffg2l_3de1980f-7b67-4fd1-9981-4f952b4ee8b0/console-operator/3.log" Jul 24 23:38:27.216853 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:27.216760 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" event={"ID":"39dead86-d402-4c70-9615-4525ba1588fd","Type":"ContainerStarted","Data":"4a9a8cf589a491aaeef039d0650e1715402125056f9e8dbd37856e54ae9c3558"} Jul 24 23:38:27.216853 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:27.216822 2571 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" event={"ID":"39dead86-d402-4c70-9615-4525ba1588fd","Type":"ContainerStarted","Data":"1fd10153fdd7743cc3112d03d852ce344e326cf92ab4d96d6b91078139384bf8"} Jul 24 23:38:27.216853 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:27.216853 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:27.232538 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:27.232491 2571 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" podStartSLOduration=2.232477211 podStartE2EDuration="2.232477211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:27.230414245 +0000 UTC m=+3136.159753449" watchObservedRunningTime="2026-07-24 23:38:27.232477211 +0000 UTC m=+3136.161816436" Jul 24 23:38:27.899612 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:27.899576 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-8sjlf_8b4a9c30-aa60-4698-be16-56f1c25a2982/volume-data-source-validator/0.log" Jul 24 23:38:28.860371 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:28.860344 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cl5g_2e9c68a4-54dd-4b07-83b4-79cb055eb539/dns/0.log" Jul 24 23:38:28.883262 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:28.883234 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7cl5g_2e9c68a4-54dd-4b07-83b4-79cb055eb539/kube-rbac-proxy/0.log" Jul 24 23:38:29.043907 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:29.043877 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lzdj7_dae193ef-02ba-4de6-bc10-c521f0ee99e9/dns-node-resolver/0.log" Jul 24 23:38:29.703667 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:29.703635 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tzrc7_bf4d6d27-e179-4ea2-b8fb-0172c057f2dd/node-ca/0.log" Jul 24 23:38:30.739528 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:30.739496 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-f7q7t_9cff92f8-a35f-411a-8cca-4155434c3ed9/kube-auth-proxy/0.log" Jul 24 23:38:30.868408 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:30.868373 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cfnx4_47034fdc-50f0-45b9-9ca7-354ebce2ec89/payload-pre-processing/0.log" Jul 24 23:38:30.903586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:30.903550 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:38:30.929152 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:30.929122 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-88d7b5c95-jqkth_5eaa6fba-900c-4fb8-b51c-821f7112fd03/router/0.log" Jul 24 23:38:31.727459 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:31.727426 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mk84c_f5d43aaf-5cc3-4545-8815-b91e60c4951f/serve-healthcheck-canary/0.log" Jul 24 23:38:32.211588 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:32.211548 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mp6t4_69cd7378-d549-481c-b80a-5f5f7e140c51/insights-operator/0.log" Jul 24 23:38:32.213264 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:32.213238 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-mp6t4_69cd7378-d549-481c-b80a-5f5f7e140c51/insights-operator/1.log" Jul 24 23:38:32.232859 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:32.232829 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c68zt_d75bbedf-d9ba-4fd1-94c2-0349016e8c2a/kube-rbac-proxy/0.log" Jul 24 23:38:32.253191 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:32.253170 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c68zt_d75bbedf-d9ba-4fd1-94c2-0349016e8c2a/exporter/0.log" Jul 24 23:38:32.276822 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:32.276801 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c68zt_d75bbedf-d9ba-4fd1-94c2-0349016e8c2a/extractor/0.log" Jul 24 23:38:33.232466 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:33.232438 2571 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qntww/perf-node-gather-daemonset-j76dg" Jul 24 23:38:35.060672 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:35.060634 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-9dbb4674-s44gl_524fb69a-b2e2-4838-8e37-c4fbe1270a5f/manager/0.log" Jul 24 23:38:35.263661 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:35.263628 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-9fbmj_1066e233-706a-4088-a4fe-fbd5a7aac431/manager/0.log" Jul 24 23:38:35.284162 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:35.284135 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d99768bd-9fbmj_1066e233-706a-4088-a4fe-fbd5a7aac431/copy-manifests/0.log" Jul 24 23:38:36.536081 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:36.536052 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-f7q7t_9cff92f8-a35f-411a-8cca-4155434c3ed9/kube-auth-proxy/0.log" Jul 24 23:38:36.637970 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:36.637937 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cfnx4_47034fdc-50f0-45b9-9ca7-354ebce2ec89/payload-pre-processing/0.log" Jul 24 23:38:36.664291 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:36.664253 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:38:36.681695 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:36.681654 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-88d7b5c95-jqkth_5eaa6fba-900c-4fb8-b51c-821f7112fd03/router/0.log" Jul 24 23:38:36.690268 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:36.690242 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-86cd477748-6cxbb_c178989e-b63b-4bfb-a6d2-06c8bc14943b/manager/0.log" Jul 24 23:38:38.232323 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:38.232234 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-849489fbdc-f7q7t_9cff92f8-a35f-411a-8cca-4155434c3ed9/kube-auth-proxy/0.log" Jul 24 23:38:38.339462 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:38.339432 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-cfnx4_47034fdc-50f0-45b9-9ca7-354ebce2ec89/payload-pre-processing/0.log" Jul 24 23:38:38.369031 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:38.368999 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-gczwg_005b637c-a677-4214-889f-772b1f919ccf/payload-processing/0.log" Jul 24 23:38:38.386618 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:38.386579 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-88d7b5c95-jqkth_5eaa6fba-900c-4fb8-b51c-821f7112fd03/router/0.log" Jul 24 23:38:42.034279 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:42.034240 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2_5ab34ede-a8e4-4faa-8447-c3ae1b89b77c/kube-storage-version-migrator-operator/1.log" Jul 24 23:38:42.035767 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:42.035725 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-vplx2_5ab34ede-a8e4-4faa-8447-c3ae1b89b77c/kube-storage-version-migrator-operator/0.log" Jul 24 23:38:43.324313 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.324282 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/kube-multus-additional-cni-plugins/0.log" Jul 24 23:38:43.345179 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.345147 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/egress-router-binary-copy/0.log" Jul 24 23:38:43.366126 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.366098 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/cni-plugins/0.log" Jul 24 23:38:43.386383 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.386356 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/bond-cni-plugin/0.log" Jul 24 23:38:43.406242 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.406210 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/routeoverride-cni/0.log" Jul 24 23:38:43.427265 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.427231 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/whereabouts-cni-bincopy/0.log" Jul 24 23:38:43.447425 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.447378 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wn7sf_9262413d-ed89-49ec-953f-dd286332ff8a/whereabouts-cni/0.log" Jul 24 23:38:43.490901 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.490864 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-dzzpw_3a27c5bb-ee99-4d93-add6-e0be2ac4200c/kube-multus/0.log" Jul 24 23:38:43.514744 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.514712 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-867cq_3c870da1-f7fd-4241-a588-26b354f604fe/network-metrics-daemon/0.log" Jul 24 23:38:43.534586 ip-10-0-136-126 kubenswrapper[2571]: I0724 23:38:43.534557 2571 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-867cq_3c870da1-f7fd-4241-a588-26b354f604fe/kube-rbac-proxy/0.log"