Jul 29 18:34:42.662796 ip-10-0-129-174 systemd[1]: Starting Kubernetes Kubelet... Jul 29 18:34:43.062901 ip-10-0-129-174 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:43.062901 ip-10-0-129-174 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 18:34:43.062901 ip-10-0-129-174 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:43.062901 ip-10-0-129-174 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 18:34:43.062901 ip-10-0-129-174 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:43.064059 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.062954 2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 18:34:43.067306 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067290 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.067306 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067305 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067310 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067314 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067317 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067320 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067323 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067326 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067328 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067331 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067334 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067336 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067339 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067347 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067350 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067352 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067355 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067357 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067361 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067365 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067368 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.067371 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067371 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067373 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067376 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067379 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067382 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067385 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067388 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067390 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067393 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067395 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067398 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067400 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067403 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067407 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067410 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067413 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067416 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067419 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067422 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067424 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.067862 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067427 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067429 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067431 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067434 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067437 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067439 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067441 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067444 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067446 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067449 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067452 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067454 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067457 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067459 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067463 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067465 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067468 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067470 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067473 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.068379 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067475 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067478 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067480 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067482 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067486 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067490 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067493 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067495 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067498 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067500 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067503 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067506 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067508 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067511 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067513 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067516 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067518 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067521 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067523 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.068868 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067526 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067528 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067531 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067533 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067535 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067538 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067554 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067556 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067561 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067564 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067567 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067569 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067572 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067574 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067578 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067580 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067583 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067586 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067588 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.069307 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.067591 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.069915 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069902 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.069915 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069914 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069919 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069922 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069926 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069929 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069932 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069934 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069937 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069939 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069942 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069944 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069947 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069949 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069952 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069954 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069957 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069959 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069962 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069964 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.069971 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069967 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069969 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069973 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069977 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069980 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069984 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069988 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069991 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069993 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069996 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.069998 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070000 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070003 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070005 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070008 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070010 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070013 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070015 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070018 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.070477 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070021 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070024 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070027 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070029 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070031 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070034 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070036 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070039 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070041 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070043 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070046 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070048 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070051 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070053 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070055 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070059 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070062 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070066 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070069 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.070962 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070072 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070076 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070078 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070081 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070084 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070086 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070088 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070091 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070093 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070096 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070098 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070101 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070103 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070106 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070109 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070112 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070115 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070117 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070120 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070122 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.071423 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070124 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070127 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070129 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070132 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070134 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070137 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070139 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070141 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070145 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070148 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070151 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070153 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070155 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070158 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070160 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070163 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070165 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070167 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070171 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070173 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.071911 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070176 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070560 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070565 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070568 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070571 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070574 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070576 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070579 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070582 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070584 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070587 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070589 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070591 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070594 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070597 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070599 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070602 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070604 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070606 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070609 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.072394 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070613 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070615 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070618 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070621 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070623 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070626 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070628 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070631 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070634 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070636 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070639 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070641 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070645 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070650 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070653 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070655 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070658 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070660 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070663 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.072909 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070665 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070668 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070670 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070673 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070675 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070678 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070680 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070682 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070685 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070687 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070690 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070693 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070696 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070698 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070701 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070703 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070706 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070709 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070711 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070714 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.073382 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070716 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070719 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070721 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070724 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070726 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070728 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070731 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070733 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070736 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070738 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070741 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070743 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070746 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070749 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070751 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070754 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070756 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070758 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070761 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.073888 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070765 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070768 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070771 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070774 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070776 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070779 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070781 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070784 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070787 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070789 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070791 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070794 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070796 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070799 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070801 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070804 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070806 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070808 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070811 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070813 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.074346 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070816 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.070819 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070891 2578 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070898 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070904 2578 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070908 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070913 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070916 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070920 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070924 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070928 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070930 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070934 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070937 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070940 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070943 2578 flags.go:64] FLAG: --cgroup-root="" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070946 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070950 2578 flags.go:64] FLAG: --client-ca-file="" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070953 2578 flags.go:64] FLAG: --cloud-provider="external" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070955 2578 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070959 2578 flags.go:64] FLAG: --cluster-domain="" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070962 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070965 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 18:34:43.074846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070968 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070972 2578 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070975 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070978 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070981 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070985 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070987 2578 flags.go:64] FLAG: --contention-profiling="false" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070990 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070993 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070996 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.070999 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071004 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071007 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071010 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071013 2578 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071016 2578 flags.go:64] FLAG: --enable-server="true" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071020 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071024 2578 flags.go:64] FLAG: --event-burst="100" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071027 2578 flags.go:64] FLAG: --event-qps="50" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071030 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071033 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071036 2578 flags.go:64] FLAG: --eviction-hard="" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071039 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071042 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071045 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 18:34:43.075395 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071048 2578 flags.go:64] FLAG: --eviction-soft="" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071052 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071055 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071058 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071061 2578 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071064 2578 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071066 2578 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071069 2578 flags.go:64] FLAG: --feature-gates="" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071073 2578 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071076 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071079 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071082 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071085 2578 flags.go:64] FLAG: --healthz-port="10248" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071088 2578 flags.go:64] FLAG: --help="false" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071090 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071094 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071097 2578 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071099 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071105 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071108 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071111 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071113 2578 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071116 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 18:34:43.076053 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071119 2578 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071122 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071125 2578 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071128 2578 flags.go:64] FLAG: --kube-reserved="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071131 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071134 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071136 2578 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071139 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071142 2578 flags.go:64] FLAG: --lock-file="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071145 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071149 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071152 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071157 2578 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071160 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071163 2578 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071165 2578 flags.go:64] FLAG: --logging-format="text" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071168 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071171 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071174 2578 flags.go:64] FLAG: --manifest-url="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071176 2578 flags.go:64] FLAG: --manifest-url-header="" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071180 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071183 2578 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071191 2578 flags.go:64] FLAG: --max-pods="110" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071194 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071197 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 18:34:43.076726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071200 2578 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071202 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071206 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071209 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071212 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071219 2578 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071222 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071225 2578 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071228 2578 flags.go:64] FLAG: --pod-cidr="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071231 2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071236 2578 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071239 2578 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071242 2578 flags.go:64] FLAG: --pods-per-core="0" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071245 2578 flags.go:64] FLAG: --port="10250" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071247 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071250 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01cfecf36d95cf334" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071253 2578 flags.go:64] FLAG: --qos-reserved="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071256 2578 flags.go:64] FLAG: --read-only-port="10255" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071260 2578 flags.go:64] FLAG: --register-node="true" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071262 2578 flags.go:64] FLAG: --register-with-taints="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071266 2578 flags.go:64] FLAG: --registry-burst="10" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071268 2578 flags.go:64] FLAG: --registry-qps="5" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071271 2578 flags.go:64] FLAG: --reserved-cpus="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071274 2578 flags.go:64] FLAG: --reserved-memory="" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071277 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 18:34:43.077347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071280 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071283 2578 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071286 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071289 2578 flags.go:64] FLAG: --runonce="false" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071291 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071294 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071297 2578 flags.go:64] FLAG: --seccomp-default="false" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071300 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071302 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071306 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071309 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071312 2578 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071315 2578 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071318 2578 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071321 2578 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071323 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071326 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071330 2578 flags.go:64] FLAG: --system-cgroups="" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071332 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071338 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071341 2578 flags.go:64] FLAG: --tls-cert-file="" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071343 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071347 2578 flags.go:64] FLAG: --tls-min-version="" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071350 2578 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071352 2578 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 18:34:43.078012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071356 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071359 2578 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071362 2578 flags.go:64] FLAG: --v="2" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071365 2578 flags.go:64] FLAG: --version="false" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071369 2578 flags.go:64] FLAG: --vmodule="" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071372 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.071375 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071487 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071491 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071494 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071496 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071499 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071502 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071505 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071507 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071511 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071515 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071517 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071520 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071523 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071527 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.078633 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071531 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071535 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071539 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071554 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071558 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071561 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071565 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071569 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071572 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071574 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071577 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071581 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071583 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071586 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071588 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071591 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071593 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071596 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071598 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.079143 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071601 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071603 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071606 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071608 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071611 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071614 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071616 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071619 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071623 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071625 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071628 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071630 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071633 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071635 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071638 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071641 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071643 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071646 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071649 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071651 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.079646 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071654 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071656 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071659 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071662 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071665 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071667 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071670 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071672 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071675 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071677 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071680 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071683 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071685 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071688 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071690 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071693 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071696 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071698 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071701 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071703 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.080142 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071707 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071709 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071712 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071714 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071717 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071720 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071722 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071724 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071727 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071729 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071732 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071735 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071738 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071740 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071743 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071745 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071749 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071752 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071754 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071757 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.080685 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071759 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071762 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071764 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071767 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071769 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.071771 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.072417 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.079907 2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.079920 2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.080689 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.080701 2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080758 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080762 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080766 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080769 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.081168 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080771 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080774 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080777 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080779 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080782 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080785 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080788 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080791 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080795 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080798 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080801 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080803 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080806 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080808 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080812 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080814 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080817 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080819 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080822 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.081553 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080824 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080827 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080829 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080831 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080834 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080837 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080840 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080843 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080845 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080848 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080850 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080853 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080855 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080858 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080860 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080864 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080868 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080871 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080873 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080876 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.082017 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080879 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080881 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080884 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080887 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080890 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080892 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080895 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080897 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080901 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080904 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080907 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080909 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080912 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080914 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080917 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080919 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080922 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080925 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080927 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080930 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.082494 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080932 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080935 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080937 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080940 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080942 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080944 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080947 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080949 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080951 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080954 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080956 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080960 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080963 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080966 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080968 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080972 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080974 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080977 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080980 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.082982 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080982 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080985 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080987 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080990 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080992 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080995 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.080998 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081000 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081003 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081005 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081007 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081010 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081013 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081015 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081018 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081020 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.083441 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081023 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.081027 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081140 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081144 2578 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081147 2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081149 2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081152 2578 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081154 2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081157 2578 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081159 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081162 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081164 2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081166 2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081169 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081172 2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:43.083848 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081174 2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081177 2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081179 2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081182 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081184 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081187 2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081189 2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081191 2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081194 2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081196 2578 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081198 2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081201 2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081203 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081206 2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081210 2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081213 2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081216 2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081219 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081221 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:43.084215 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081223 2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081226 2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081230 2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081233 2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081236 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081239 2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081241 2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081244 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081246 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081249 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081251 2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081253 2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081256 2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081259 2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081261 2578 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081264 2578 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081266 2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081268 2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081271 2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:43.084727 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081273 2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081276 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081278 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081280 2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081283 2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081285 2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081288 2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081291 2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081293 2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081296 2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081298 2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081301 2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081303 2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081305 2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081308 2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081310 2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081312 2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081315 2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081317 2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081320 2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:43.085214 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081322 2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081324 2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081327 2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081329 2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081332 2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081334 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081337 2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081340 2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081342 2578 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081344 2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081347 2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081349 2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081352 2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081354 2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081356 2578 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081359 2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081361 2578 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081364 2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081366 2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081369 2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:43.085894 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081371 2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081374 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081377 2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081380 2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081383 2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081386 2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081389 2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:43.081391 2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.081396 2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.081496 2578 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.084169 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.085428 2578 server.go:1019] "Starting client certificate rotation" Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.085557 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:43.086378 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.085597 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:43.087843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.087827 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:43.093214 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.093192 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:43.106259 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.106243 2578 log.go:25] "Validated CRI v1 runtime API" Jul 29 18:34:43.108187 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.108168 2578 log.go:25] "Validated CRI v1 image API" Jul 29 18:34:43.109047 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.109024 2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 18:34:43.114009 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.113976 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9088db8f-ca05-49fb-a03a-cebe435d838e:/dev/nvme0n1p3 e0b35dd7-78f9-4c67-9141-c0db3d68f73e:/dev/nvme0n1p4] Jul 29 18:34:43.114088 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.114009 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 18:34:43.117918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.117893 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:43.119318 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.119212 2578 manager.go:216] Machine: {Timestamp:2026-07-29 18:34:43.117936404 +0000 UTC m=+0.333101424 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100256 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2dd905792b67393188b37981290ad3 SystemUUID:ec2dd905-792b-6739-3188-b37981290ad3 BootID:80749b81-cdd9-4bec-982e-87973cddd7f1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a8:a9:f5:16:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a8:a9:f5:16:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:79:fd:8a:a5:5a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 18:34:43.119318 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.119315 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 18:34:43.119427 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.119389 2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 18:34:43.122166 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.122144 2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 18:34:43.122294 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.122170 2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-174.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 18:34:43.123506 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.123495 2578 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 18:34:43.123565 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.123507 2578 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 18:34:43.123565 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.123515 2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:43.124303 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.124292 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:43.125093 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.125083 2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 18:34:43.127001 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.126990 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:43.127108 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127099 2578 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 18:34:43.127148 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127127 2578 kubelet.go:486] "Attempting to sync node with API server" Jul 29 18:34:43.127148 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127144 2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 18:34:43.127205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127155 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 18:34:43.127205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127163 2578 kubelet.go:398] "Adding apiserver pod source" Jul 29 18:34:43.127205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127172 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 18:34:43.127556 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127522 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:43.127619 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.127563 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:43.131148 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.131129 2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 18:34:43.133062 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.133049 2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 18:34:43.133113 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.133079 2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 18:34:43.134479 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134465 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 18:34:43.134523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134488 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 18:34:43.134523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134497 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 18:34:43.134523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134505 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 18:34:43.134523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134517 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 18:34:43.134661 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134525 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 18:34:43.134661 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134537 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 18:34:43.134661 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134565 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 18:34:43.134661 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134574 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 18:34:43.134797 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134738 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 18:34:43.134797 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134771 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 18:34:43.134797 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.134783 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 18:34:43.135675 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.135655 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 18:34:43.135763 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.135686 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 18:34:43.137866 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.137845 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 18:34:43.138045 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.138032 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-174.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 18:34:43.138110 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.138092 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 18:34:43.139386 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.139375 2578 server.go:1268] "Started kubelet" Jul 29 18:34:43.139624 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.139519 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 18:34:43.139624 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.139569 2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 18:34:43.139761 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.139641 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 18:34:43.140145 ip-10-0-129-174 systemd[1]: Started Kubernetes Kubelet. Jul 29 18:34:43.141325 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.141307 2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 18:34:43.141795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.141782 2578 server.go:310] "Adding debug handlers to kubelet server" Jul 29 18:34:43.142067 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.142050 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nk2s" Jul 29 18:34:43.145773 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.145746 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 18:34:43.147705 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.147686 2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 18:34:43.147775 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.147688 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:43.147816 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.147787 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.148292 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148273 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 18:34:43.148292 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148277 2578 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 18:34:43.148418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148306 2578 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 18:34:43.148418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148392 2578 reconstruct.go:97] "Volume reconstruction finished" Jul 29 18:34:43.148418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148402 2578 reconciler.go:29] "Reconciler: start to sync state" Jul 29 18:34:43.148673 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148657 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 18:34:43.148673 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148674 2578 factory.go:55] Registering systemd factory Jul 29 18:34:43.148813 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148684 2578 factory.go:223] Registration of the systemd container factory successfully Jul 29 18:34:43.148903 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148886 2578 factory.go:153] Registering CRI-O factory Jul 29 18:34:43.148903 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148904 2578 factory.go:223] Registration of the crio container factory successfully Jul 29 18:34:43.149031 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148923 2578 factory.go:103] Registering Raw factory Jul 29 18:34:43.149031 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.148933 2578 manager.go:1196] Started watching for new ooms in manager Jul 29 18:34:43.149314 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.149303 2578 manager.go:349] Starting recovery of all containers Jul 29 18:34:43.150124 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.150106 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7nk2s" Jul 29 18:34:43.154085 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.154060 2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 18:34:43.154486 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.154462 2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-174.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 18:34:43.155276 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.154196 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-174.ec2.internal.18c6d671013c2430 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-174.ec2.internal,UID:ip-10-0-129-174.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-174.ec2.internal,},FirstTimestamp:2026-07-29 18:34:43.139347504 +0000 UTC m=+0.354512524,LastTimestamp:2026-07-29 18:34:43.139347504 +0000 UTC m=+0.354512524,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-174.ec2.internal,}" Jul 29 18:34:43.161834 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.161818 2578 manager.go:354] Recovery completed Jul 29 18:34:43.165933 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.165920 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.167963 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.167937 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.168057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.167983 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.168057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.168000 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.168932 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.168913 2578 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 18:34:43.168932 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.168933 2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 18:34:43.169057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.168960 2578 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:43.169057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.168974 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.169936 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.169917 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.170048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.170028 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.170143 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.170065 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.170632 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.170620 2578 policy_none.go:49] "None policy: Start" Jul 29 18:34:43.170700 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.170637 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 18:34:43.170700 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.170648 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 18:34:43.171592 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.171582 2578 policy_none.go:47] "Start" Jul 29 18:34:43.212244 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212075 2578 manager.go:337] "Starting Device Plugin manager" Jul 29 18:34:43.212320 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.212272 2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 18:34:43.212320 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212288 2578 server.go:87] "Starting device plugin registration server" Jul 29 18:34:43.212500 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212487 2578 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 18:34:43.212570 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212501 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 18:34:43.212634 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212605 2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 18:34:43.212731 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212696 2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 18:34:43.212731 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.212709 2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 18:34:43.213355 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.213333 2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 18:34:43.213412 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.213381 2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.312870 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.312847 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.313883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.313751 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.313883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.313785 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.313883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.313796 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.313883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.313822 2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.323171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.323157 2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.323219 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.323177 2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-174.ec2.internal\": node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.346379 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.346347 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 18:34:43.347936 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.347914 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 18:34:43.348036 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.347957 2578 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 18:34:43.348036 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.347982 2578 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 18:34:43.348036 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.348020 2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 18:34:43.350072 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.350056 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:43.353407 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.353387 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.449144 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.449112 2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal"] Jul 29 18:34:43.449268 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.449195 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.450788 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.450771 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.450833 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.450807 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.450833 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.450820 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.451979 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.451967 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.452120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.452170 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452136 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.452713 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452691 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.452819 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452726 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.452819 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452737 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.452819 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452691 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.452819 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452806 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.452966 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.452824 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.453460 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.453446 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.453916 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.453904 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.453958 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.453928 2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:43.454616 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.454600 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:43.454681 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.454631 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:43.454681 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.454645 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:43.475114 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.475093 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-174.ec2.internal\" not found" node="ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.479780 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.479764 2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-174.ec2.internal\" not found" node="ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.549651 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.549623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb52c0df565696781869de4afb3c91b0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-174.ec2.internal\" (UID: \"eb52c0df565696781869de4afb3c91b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.549651 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.549653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.549844 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.549669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.553620 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.553601 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.650470 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb52c0df565696781869de4afb3c91b0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-174.ec2.internal\" (UID: \"eb52c0df565696781869de4afb3c91b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.650470 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.650470 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.650688 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650512 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.650688 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650513 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb52c0df565696781869de4afb3c91b0-config\") pod \"kube-apiserver-proxy-ip-10-0-129-174.ec2.internal\" (UID: \"eb52c0df565696781869de4afb3c91b0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.650688 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.650536 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c30e97374955d8fd85f1ca8568883e75-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal\" (UID: \"c30e97374955d8fd85f1ca8568883e75\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.654481 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.654465 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.754968 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.754940 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.777099 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.777058 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.782877 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:43.782857 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:43.855828 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.855780 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:43.956439 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:43.956353 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.056687 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.056656 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.086170 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.086141 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 18:34:44.086628 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.086320 2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 18:34:44.155787 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.155743 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:29:43 +0000 UTC" deadline="2028-02-03 10:25:58.430209321 +0000 UTC" Jul 29 18:34:44.155787 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.155781 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13287h51m14.274433258s" Jul 29 18:34:44.156774 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.156743 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.257727 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.257640 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.358692 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.358657 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.399645 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:44.399577 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc30e97374955d8fd85f1ca8568883e75.slice/crio-2bb923ef2a10de5adebd83b64fa64d38e6ddf7f7bfb8dd0feff0b09ab99ec3af WatchSource:0}: Error finding container 2bb923ef2a10de5adebd83b64fa64d38e6ddf7f7bfb8dd0feff0b09ab99ec3af: Status 404 returned error can't find the container with id 2bb923ef2a10de5adebd83b64fa64d38e6ddf7f7bfb8dd0feff0b09ab99ec3af Jul 29 18:34:44.399852 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:44.399831 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb52c0df565696781869de4afb3c91b0.slice/crio-ce4a5176703b6c9fc64ca2054e0bf80f14fdee3971c18827a7de9a8f79ea934a WatchSource:0}: Error finding container ce4a5176703b6c9fc64ca2054e0bf80f14fdee3971c18827a7de9a8f79ea934a: Status 404 returned error can't find the container with id ce4a5176703b6c9fc64ca2054e0bf80f14fdee3971c18827a7de9a8f79ea934a Jul 29 18:34:44.404769 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.404749 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:34:44.431190 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.431154 2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:44.458813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.458763 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.519132 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.519044 2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:44.559346 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.559313 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.659621 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:44.659581 2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-174.ec2.internal\" not found" Jul 29 18:34:44.692289 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.692256 2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:44.748339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.748307 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" Jul 29 18:34:44.806442 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.806412 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:44.807656 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.807629 2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" Jul 29 18:34:44.814913 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:44.814885 2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:45.072532 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.072456 2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:45.129135 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.129097 2578 apiserver.go:52] "Watching apiserver" Jul 29 18:34:45.140029 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.139996 2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 18:34:45.140530 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.140501 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-wdk2w","openshift-multus/network-metrics-daemon-q4h9l","kube-system/global-pull-secret-syncer-2vwxh","kube-system/konnectivity-agent-dbhmw","kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal","openshift-network-diagnostics/network-check-target-qxz79","openshift-network-operator/iptables-alerter-qz6nh","openshift-ovn-kubernetes/ovnkube-node-jrhmw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr","openshift-cluster-node-tuning-operator/tuned-2xg8x","openshift-dns/node-resolver-wpmjn","openshift-image-registry/node-ca-tr8pb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal","openshift-multus/multus-additional-cni-plugins-t68xd"] Jul 29 18:34:45.142329 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.142308 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.143646 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.143619 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.144842 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.144821 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.144948 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.144903 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:45.146248 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.146221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.147718 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.147695 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.148646 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.148624 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:45.148770 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.148695 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:45.148770 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.148756 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:45.150293 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.150271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.150513 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.150493 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 18:34:45.151106 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.151088 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-pdt4b\"" Jul 29 18:34:45.151819 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.151799 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.153147 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.153127 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.153243 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.153200 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:45.154670 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154225 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 18:34:45.154670 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154257 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 18:34:45.154670 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.154840 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 18:34:45.154840 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154824 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.155196 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154969 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 18:34:45.155291 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155274 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 18:34:45.155495 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155477 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.155631 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155602 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qj7qr\"" Jul 29 18:34:45.155777 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155761 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 18:34:45.155847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155793 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.155912 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.154765 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.155912 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.155484 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.156033 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156014 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 18:34:45.156089 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156059 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.156150 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156132 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnn6z\"" Jul 29 18:34:45.156242 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156161 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-d5zlt\"" Jul 29 18:34:45.156242 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156017 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.156358 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156334 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nlvld\"" Jul 29 18:34:45.156358 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156341 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 18:34:45.156478 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156062 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 18:34:45.156833 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156799 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 18:34:45.156928 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156887 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kd5rp\"" Jul 29 18:34:45.156986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.156969 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 18:34:45.159492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.159474 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.160498 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160474 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-var-lib-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160628 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-etc-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160628 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160538 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160628 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160563 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e637449e-59bf-41ce-a871-63f3c81263fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160628 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160590 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cnibin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160633 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-iptables-alerter-script\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160669 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-kubelet\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160694 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160725 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-system-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160751 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-hostroot\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-host-slash\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160802 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlgd2\" (UniqueName: \"kubernetes.io/projected/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-kube-api-access-jlgd2\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160827 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-netns\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.160859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-log-socket\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160907 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-netd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160936 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-env-overrides\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160961 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-script-lib\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.160978 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161003 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8k6\" (UniqueName: \"kubernetes.io/projected/e637449e-59bf-41ce-a871-63f3c81263fe-kube-api-access-rk8k6\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161030 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-socket-dir-parent\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161054 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-netns\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161087 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161134 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33461b5b-deeb-4b99-af20-390ad7ccdac6-agent-certs\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cni-binary-copy\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-conf-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-etc-kubernetes\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-systemd-units\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161422 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-slash\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-ovn\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-os-release\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.161795 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161783 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-multus\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161828 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-systemd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-dbus\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.161949 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-bin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162007 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-bin\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-config\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.162140 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162139 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-kubelet-config\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162164 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33461b5b-deeb-4b99-af20-390ad7ccdac6-konnectivity-ca\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162188 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162223 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-k8s-cni-cncf-io\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162265 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-node-log\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-kubelet\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-daemon-config\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-multus-certs\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttwx2\" (UniqueName: \"kubernetes.io/projected/afd0e76f-9dd0-429b-95ad-214ee9962d1d-kube-api-access-ttwx2\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.162492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.162405 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:45.163356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.163338 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.163718 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.163690 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.164008 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.163827 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9g2zt\"" Jul 29 18:34:45.164706 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.164204 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.164706 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.164343 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.165360 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.164997 2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:45.165360 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.165004 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.165360 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.165252 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-47sht\"" Jul 29 18:34:45.168422 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.168401 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 18:34:45.169081 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.169059 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 18:34:45.169415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.169396 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 18:34:45.169747 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.169727 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zpgb4\"" Jul 29 18:34:45.190953 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.190918 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcb2q" Jul 29 18:34:45.200625 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.200591 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wcb2q" Jul 29 18:34:45.249857 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.249819 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 18:34:45.262846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33461b5b-deeb-4b99-af20-390ad7ccdac6-konnectivity-ca\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.262846 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262848 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-k8s-cni-cncf-io\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262896 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-daemon-config\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262917 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-node-log\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262939 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttwx2\" (UniqueName: \"kubernetes.io/projected/afd0e76f-9dd0-429b-95ad-214ee9962d1d-kube-api-access-ttwx2\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262966 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2twc8\" (UniqueName: \"kubernetes.io/projected/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kube-api-access-2twc8\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.262993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-modprobe-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263036 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-run\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263060 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-etc-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.263097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263099 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e637449e-59bf-41ce-a871-63f3c81263fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263124 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cnibin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263155 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263181 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-var-lib-kubelet\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-kubelet\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-hostroot\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-host-slash\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263294 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-conf\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-lib-modules\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263352 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtdjn\" (UniqueName: \"kubernetes.io/projected/703e1173-0804-4069-8845-99e75777cd3b-kube-api-access-xtdjn\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-netns\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263406 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-device-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263485 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysconfig\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-cnibin\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263573 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33461b5b-deeb-4b99-af20-390ad7ccdac6-agent-certs\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.263724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263600 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cni-binary-copy\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-etc-kubernetes\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-registration-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263702 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-host\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263744 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/cf0d1aa8-2165-473b-a77d-25e2af93723a-kube-api-access-zcsqs\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263784 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-ovn\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-os-release\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-multus\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-sys\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-systemd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-dbus\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.263970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-bin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25n28\" (UniqueName: \"kubernetes.io/projected/28edf3c8-c9a6-4b2f-a826-7644923b3345-kube-api-access-25n28\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-systemd\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-config\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-kubelet\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264190 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-multus-certs\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.264566 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264332 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264390 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-var-lib-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264448 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-iptables-alerter-script\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-socket-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-tuned\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264530 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-tmp\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264559 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-system-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlgd2\" (UniqueName: \"kubernetes.io/projected/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-kube-api-access-jlgd2\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/703e1173-0804-4069-8845-99e75777cd3b-host\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264732 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/703e1173-0804-4069-8845-99e75777cd3b-serviceca\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.265432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rz7\" (UniqueName: \"kubernetes.io/projected/5730df58-4eca-443a-8bbf-31ffe7667927-kube-api-access-77rz7\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-netns\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264818 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-log-socket\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264875 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-netd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-env-overrides\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-script-lib\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8k6\" (UniqueName: \"kubernetes.io/projected/e637449e-59bf-41ce-a871-63f3c81263fe-kube-api-access-rk8k6\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.264996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-socket-dir-parent\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265026 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-conf-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265056 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-sys-fs\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265085 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-system-cni-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-binary-copy\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5730df58-4eca-443a-8bbf-31ffe7667927-hosts-file\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265182 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5730df58-4eca-443a-8bbf-31ffe7667927-tmp-dir\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-systemd-units\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-slash\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265271 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-kubernetes\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.266274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-os-release\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvgpj\" (UniqueName: \"kubernetes.io/projected/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-kube-api-access-tvgpj\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265400 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-bin\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265430 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-kubelet-config\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.265553 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-kubelet-config\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.266255 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33461b5b-deeb-4b99-af20-390ad7ccdac6-konnectivity-ca\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.266515 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-k8s-cni-cncf-io\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.266545 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.267090 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267082 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-daemon-config\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267140 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-kubelet\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267187 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-multus-certs\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267198 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-config\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267266 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-node-log\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267517 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-etc-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267549 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-var-lib-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.267663 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267608 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-openvswitch\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.268004 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.267885 2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 18:34:45.268162 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268112 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-iptables-alerter-script\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.268308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268250 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.268308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268286 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-kubelet\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.268308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268289 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-os-release\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.268308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268306 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cnibin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.268543 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.268345 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-hostroot\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.269234 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269210 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/afd0e76f-9dd0-429b-95ad-214ee9962d1d-cni-binary-copy\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.269404 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269377 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-multus\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.269849 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-systemd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.269921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269882 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-netns\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.269965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269924 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-netd\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.269965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-ovnkube-script-lib\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270079 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.269990 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-systemd-units\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270079 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270024 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270079 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270059 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-slash\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270096 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-run-netns\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270147 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-conf-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270161 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-host-var-lib-cni-bin\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270195 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-multus-socket-dir-parent\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-etc-kubernetes\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270333 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-host-slash\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.270462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270356 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-host-cni-bin\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-run-ovn\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.270671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.270464 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:45.270671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270467 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/afd0e76f-9dd0-429b-95ad-214ee9962d1d-system-cni-dir\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.270671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270521 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-dbus\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.270671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.270603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:45.770534501 +0000 UTC m=+2.985699526 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:45.270896 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.270691 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e637449e-59bf-41ce-a871-63f3c81263fe-log-socket\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.271667 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.271609 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e637449e-59bf-41ce-a871-63f3c81263fe-env-overrides\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.272055 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.272031 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e637449e-59bf-41ce-a871-63f3c81263fe-ovn-node-metrics-cert\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.273618 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.273593 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33461b5b-deeb-4b99-af20-390ad7ccdac6-agent-certs\") pod \"konnectivity-agent-dbhmw\" (UID: \"33461b5b-deeb-4b99-af20-390ad7ccdac6\") " pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.275589 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.275561 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:45.275589 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.275589 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:45.275746 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.275604 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:45.275746 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.275702 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:45.775681803 +0000 UTC m=+2.990846815 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:45.279372 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.279351 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttwx2\" (UniqueName: \"kubernetes.io/projected/afd0e76f-9dd0-429b-95ad-214ee9962d1d-kube-api-access-ttwx2\") pod \"multus-wdk2w\" (UID: \"afd0e76f-9dd0-429b-95ad-214ee9962d1d\") " pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.279486 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.279465 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8k6\" (UniqueName: \"kubernetes.io/projected/e637449e-59bf-41ce-a871-63f3c81263fe-kube-api-access-rk8k6\") pod \"ovnkube-node-jrhmw\" (UID: \"e637449e-59bf-41ce-a871-63f3c81263fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.280645 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.280546 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlgd2\" (UniqueName: \"kubernetes.io/projected/c9dd8bf2-7f5b-4d60-a20f-6c1251a90596-kube-api-access-jlgd2\") pod \"iptables-alerter-qz6nh\" (UID: \"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596\") " pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.355192 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.355060 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" event={"ID":"c30e97374955d8fd85f1ca8568883e75","Type":"ContainerStarted","Data":"2bb923ef2a10de5adebd83b64fa64d38e6ddf7f7bfb8dd0feff0b09ab99ec3af"} Jul 29 18:34:45.356561 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.356502 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" event={"ID":"eb52c0df565696781869de4afb3c91b0","Type":"ContainerStarted","Data":"ce4a5176703b6c9fc64ca2054e0bf80f14fdee3971c18827a7de9a8f79ea934a"} Jul 29 18:34:45.366064 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366030 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2twc8\" (UniqueName: \"kubernetes.io/projected/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kube-api-access-2twc8\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366073 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-modprobe-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366121 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-run\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366147 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366171 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-var-lib-kubelet\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366209 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-conf\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366239 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-lib-modules\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtdjn\" (UniqueName: \"kubernetes.io/projected/703e1173-0804-4069-8845-99e75777cd3b-kube-api-access-xtdjn\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-device-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366314 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysconfig\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366367 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-cnibin\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-registration-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366440 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-host\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/cf0d1aa8-2165-473b-a77d-25e2af93723a-kube-api-access-zcsqs\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366498 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-modprobe-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-sys\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366553 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25n28\" (UniqueName: \"kubernetes.io/projected/28edf3c8-c9a6-4b2f-a826-7644923b3345-kube-api-access-25n28\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366574 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-systemd\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.366633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-sys\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-lib-modules\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366840 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-d\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366847 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-systemd\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366890 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-run\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366914 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-var-lib-kubelet\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366944 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.366968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-cnibin\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367071 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysctl-conf\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367133 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-registration-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367185 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-host\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.367254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367225 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-device-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.367750 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.367312 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:45.367750 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.367388 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:45.867371778 +0000 UTC m=+3.082536790 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:45.367750 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.367750 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367457 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.367750 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367541 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-sysconfig\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.368012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.367497 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.368119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368052 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-socket-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.368165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-tuned\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.368288 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368189 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-tmp\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.368342 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368320 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.368398 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368352 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/703e1173-0804-4069-8845-99e75777cd3b-host\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.368398 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368385 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/703e1173-0804-4069-8845-99e75777cd3b-serviceca\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.368489 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rz7\" (UniqueName: \"kubernetes.io/projected/5730df58-4eca-443a-8bbf-31ffe7667927-kube-api-access-77rz7\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.368606 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-sys-fs\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.368662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-system-cni-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.368662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-binary-copy\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.368776 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5730df58-4eca-443a-8bbf-31ffe7667927-hosts-file\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.368776 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368699 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5730df58-4eca-443a-8bbf-31ffe7667927-tmp-dir\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.368885 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368859 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-kubernetes\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.369019 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368891 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-os-release\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.369019 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.368910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvgpj\" (UniqueName: \"kubernetes.io/projected/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-kube-api-access-tvgpj\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.373702 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.373923 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374016 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5730df58-4eca-443a-8bbf-31ffe7667927-hosts-file\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374197 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5730df58-4eca-443a-8bbf-31ffe7667927-tmp-dir\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374303 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-system-cni-dir\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374323 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-binary-copy\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374362 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-sys-fs\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374416 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-kubernetes\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374438 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-socket-dir\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374457 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/703e1173-0804-4069-8845-99e75777cd3b-host\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374469 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/703e1173-0804-4069-8845-99e75777cd3b-serviceca\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.374678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.374491 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cf0d1aa8-2165-473b-a77d-25e2af93723a-os-release\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.375652 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.375588 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.375764 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.375608 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf0d1aa8-2165-473b-a77d-25e2af93723a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.378086 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.377697 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-etc-tuned\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.379307 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.378655 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-tmp\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.380346 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.380319 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25n28\" (UniqueName: \"kubernetes.io/projected/28edf3c8-c9a6-4b2f-a826-7644923b3345-kube-api-access-25n28\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.380482 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.380452 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2twc8\" (UniqueName: \"kubernetes.io/projected/3d7ea662-72c5-4589-9b1a-0e04d9268ba3-kube-api-access-2twc8\") pod \"aws-ebs-csi-driver-node-v4qvr\" (UID: \"3d7ea662-72c5-4589-9b1a-0e04d9268ba3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.380993 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.380970 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtdjn\" (UniqueName: \"kubernetes.io/projected/703e1173-0804-4069-8845-99e75777cd3b-kube-api-access-xtdjn\") pod \"node-ca-tr8pb\" (UID: \"703e1173-0804-4069-8845-99e75777cd3b\") " pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.382183 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.382155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcsqs\" (UniqueName: \"kubernetes.io/projected/cf0d1aa8-2165-473b-a77d-25e2af93723a-kube-api-access-zcsqs\") pod \"multus-additional-cni-plugins-t68xd\" (UID: \"cf0d1aa8-2165-473b-a77d-25e2af93723a\") " pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.384991 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.384965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rz7\" (UniqueName: \"kubernetes.io/projected/5730df58-4eca-443a-8bbf-31ffe7667927-kube-api-access-77rz7\") pod \"node-resolver-wpmjn\" (UID: \"5730df58-4eca-443a-8bbf-31ffe7667927\") " pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.386781 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.386759 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvgpj\" (UniqueName: \"kubernetes.io/projected/4fbc4845-56bc-4ff0-b6b6-c49acddbce00-kube-api-access-tvgpj\") pod \"tuned-2xg8x\" (UID: \"4fbc4845-56bc-4ff0-b6b6-c49acddbce00\") " pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.459938 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.459893 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-qz6nh" Jul 29 18:34:45.471323 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.471289 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wdk2w" Jul 29 18:34:45.481735 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.481703 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:34:45.488945 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.488916 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:34:45.498017 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.497972 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" Jul 29 18:34:45.507300 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.507265 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t68xd" Jul 29 18:34:45.516675 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.516637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" Jul 29 18:34:45.523790 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.523754 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wpmjn" Jul 29 18:34:45.530844 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.530814 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tr8pb" Jul 29 18:34:45.772411 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.772382 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:45.772585 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.772546 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:45.772666 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.772654 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.772634552 +0000 UTC m=+3.987799565 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:45.873251 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.873212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:45.873477 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:45.873280 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:45.873477 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873427 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:45.873477 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873452 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:45.873477 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873466 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:45.873736 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873493 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:45.873736 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873532 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.873515248 +0000 UTC m=+4.088680257 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:45.873736 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:45.873549 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:46.873541722 +0000 UTC m=+4.088706731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:46.205743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.205651 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:45 +0000 UTC" deadline="2028-04-11 10:07:20.15713366 +0000 UTC" Jul 29 18:34:46.205743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.205685 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14919h32m33.951452625s" Jul 29 18:34:46.349324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.349280 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:46.349510 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.349301 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:46.349510 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.349454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:46.349673 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.349546 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:46.437391 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.437357 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0d1aa8_2165_473b_a77d_25e2af93723a.slice/crio-0431ac3f586c76eec0c5cce824b977e632081f319f42e52e689ce5db2c2db5c7 WatchSource:0}: Error finding container 0431ac3f586c76eec0c5cce824b977e632081f319f42e52e689ce5db2c2db5c7: Status 404 returned error can't find the container with id 0431ac3f586c76eec0c5cce824b977e632081f319f42e52e689ce5db2c2db5c7 Jul 29 18:34:46.445105 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.445074 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5730df58_4eca_443a_8bbf_31ffe7667927.slice/crio-f1a8a5829976050e56667c756cb596f002840147eb3c36d277e58b1297031d62 WatchSource:0}: Error finding container f1a8a5829976050e56667c756cb596f002840147eb3c36d277e58b1297031d62: Status 404 returned error can't find the container with id f1a8a5829976050e56667c756cb596f002840147eb3c36d277e58b1297031d62 Jul 29 18:34:46.446682 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.446612 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ea662_72c5_4589_9b1a_0e04d9268ba3.slice/crio-abb5a46747aba8218fde7044e590e5e272c4082f58e220e0398b298f4aa64479 WatchSource:0}: Error finding container abb5a46747aba8218fde7044e590e5e272c4082f58e220e0398b298f4aa64479: Status 404 returned error can't find the container with id abb5a46747aba8218fde7044e590e5e272c4082f58e220e0398b298f4aa64479 Jul 29 18:34:46.447625 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.447603 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33461b5b_deeb_4b99_af20_390ad7ccdac6.slice/crio-9e0f09cc5601f324a773b03fcf031280718199c1935234838fca21db0499215a WatchSource:0}: Error finding container 9e0f09cc5601f324a773b03fcf031280718199c1935234838fca21db0499215a: Status 404 returned error can't find the container with id 9e0f09cc5601f324a773b03fcf031280718199c1935234838fca21db0499215a Jul 29 18:34:46.448834 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.448810 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dd8bf2_7f5b_4d60_a20f_6c1251a90596.slice/crio-25c667d5c28dab58f7b8669634f48bc221441a26437689bf09f7c3777e83497d WatchSource:0}: Error finding container 25c667d5c28dab58f7b8669634f48bc221441a26437689bf09f7c3777e83497d: Status 404 returned error can't find the container with id 25c667d5c28dab58f7b8669634f48bc221441a26437689bf09f7c3777e83497d Jul 29 18:34:46.451688 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:34:46.451662 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703e1173_0804_4069_8845_99e75777cd3b.slice/crio-a051d6a6b690db305ffad2530b0b8ef224dfa72263063b4ad07d28dfc6339550 WatchSource:0}: Error finding container a051d6a6b690db305ffad2530b0b8ef224dfa72263063b4ad07d28dfc6339550: Status 404 returned error can't find the container with id a051d6a6b690db305ffad2530b0b8ef224dfa72263063b4ad07d28dfc6339550 Jul 29 18:34:46.782424 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.782217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:46.782424 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.782381 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:46.782669 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.782483 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.782466257 +0000 UTC m=+5.997631268 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:46.883775 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.883740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:46.883948 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:46.883809 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:46.883948 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.883931 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:46.883948 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.883934 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:46.884070 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.883957 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:46.884070 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.883971 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:46.884070 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.884010 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.88399083 +0000 UTC m=+6.099155855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:46.884070 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:46.884028 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:48.884020629 +0000 UTC m=+6.099185642 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:47.206918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.206809 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:45 +0000 UTC" deadline="2028-03-25 05:41:37.267965362 +0000 UTC" Jul 29 18:34:47.206918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.206859 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14507h6m50.061112267s" Jul 29 18:34:47.356757 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.355819 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:47.356757 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:47.356040 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:47.374802 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.374754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"780e93eee476c54b94b8fd36c9bba49222d4a8e6e701cb10b86700d51627099d"} Jul 29 18:34:47.382750 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.382693 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerStarted","Data":"0431ac3f586c76eec0c5cce824b977e632081f319f42e52e689ce5db2c2db5c7"} Jul 29 18:34:47.392362 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.392231 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbhmw" event={"ID":"33461b5b-deeb-4b99-af20-390ad7ccdac6","Type":"ContainerStarted","Data":"9e0f09cc5601f324a773b03fcf031280718199c1935234838fca21db0499215a"} Jul 29 18:34:47.405879 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.405759 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" event={"ID":"3d7ea662-72c5-4589-9b1a-0e04d9268ba3","Type":"ContainerStarted","Data":"abb5a46747aba8218fde7044e590e5e272c4082f58e220e0398b298f4aa64479"} Jul 29 18:34:47.410123 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.410035 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" event={"ID":"4fbc4845-56bc-4ff0-b6b6-c49acddbce00","Type":"ContainerStarted","Data":"32461c4dbe9a370e7cd153afffea4d50a83dd37346679521ab63e5c6e9852af8"} Jul 29 18:34:47.413922 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.413868 2578 generic.go:356] "Generic (PLEG): container finished" podID="c30e97374955d8fd85f1ca8568883e75" containerID="a865e652a427c61a86014fa9d519352f52f91877ffe7826edf8581e45dddd235" exitCode=0 Jul 29 18:34:47.414333 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.414264 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" event={"ID":"c30e97374955d8fd85f1ca8568883e75","Type":"ContainerDied","Data":"a865e652a427c61a86014fa9d519352f52f91877ffe7826edf8581e45dddd235"} Jul 29 18:34:47.419534 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.419464 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" event={"ID":"eb52c0df565696781869de4afb3c91b0","Type":"ContainerStarted","Data":"4643719afa9bd21b2ca41b7d167e8d0c2c69a2efb54198576c3bb0a323dfc73a"} Jul 29 18:34:47.422258 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.422226 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qz6nh" event={"ID":"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596","Type":"ContainerStarted","Data":"25c667d5c28dab58f7b8669634f48bc221441a26437689bf09f7c3777e83497d"} Jul 29 18:34:47.423684 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.423654 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tr8pb" event={"ID":"703e1173-0804-4069-8845-99e75777cd3b","Type":"ContainerStarted","Data":"a051d6a6b690db305ffad2530b0b8ef224dfa72263063b4ad07d28dfc6339550"} Jul 29 18:34:47.425176 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.425141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wpmjn" event={"ID":"5730df58-4eca-443a-8bbf-31ffe7667927","Type":"ContainerStarted","Data":"f1a8a5829976050e56667c756cb596f002840147eb3c36d277e58b1297031d62"} Jul 29 18:34:47.427471 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.427443 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wdk2w" event={"ID":"afd0e76f-9dd0-429b-95ad-214ee9962d1d","Type":"ContainerStarted","Data":"7b85dc2ff19fabe1085c901189ec6cd0111ee953da17e115e18b68a07f3888bc"} Jul 29 18:34:47.448461 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:47.448371 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-174.ec2.internal" podStartSLOduration=2.448344586 podStartE2EDuration="2.448344586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:47.447235711 +0000 UTC m=+4.662400736" watchObservedRunningTime="2026-07-29 18:34:47.448344586 +0000 UTC m=+4.663509627" Jul 29 18:34:48.349291 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.349247 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:48.349904 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.349417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:48.349904 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.349865 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:48.350042 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.349970 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:48.437873 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.437828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" event={"ID":"c30e97374955d8fd85f1ca8568883e75","Type":"ContainerStarted","Data":"48018bdd51c4cd4e74c679e9dd3de42a8647ab5c2a3779abd05d52d8cf63de89"} Jul 29 18:34:48.804365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.804321 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:48.804573 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.804518 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:48.804686 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.804631 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.804606089 +0000 UTC m=+10.019771104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:48.907504 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.907458 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:48.907733 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:48.907543 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:48.907813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.907737 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:48.907813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.907763 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:48.907813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.907778 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:48.908016 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.907843 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.907822778 +0000 UTC m=+10.122987790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:48.908322 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.908299 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:48.908467 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:48.908357 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:52.908339814 +0000 UTC m=+10.123504822 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:49.350669 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:49.350635 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:49.351168 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:49.350775 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:50.348514 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:50.348473 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:50.348747 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:50.348657 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:50.348747 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:50.348740 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:50.348870 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:50.348823 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:51.351274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:51.349684 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:51.351274 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:51.349856 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:52.349883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:52.349288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:52.349883 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:52.349288 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:52.349883 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.349491 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:52.349883 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.349565 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:52.843695 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:52.843643 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:52.844205 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.843814 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:52.844205 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.843887 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.843866868 +0000 UTC m=+18.059031879 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:52.944260 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:52.944313 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944444 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944461 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944471 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944517 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.944502767 +0000 UTC m=+18.159667776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944578 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:52.945003 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:52.944603 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:00.944594409 +0000 UTC m=+18.159759418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:53.350454 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:53.350423 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:53.350664 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:53.350529 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:54.349315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:54.348813 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:54.349315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:54.348849 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:54.349315 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:54.348961 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:54.349315 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:54.349062 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:55.351148 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:55.351107 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:55.351579 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:55.351249 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:56.348858 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:56.348815 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:56.348858 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:56.348860 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:56.349110 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:56.348957 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:56.349166 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:56.349104 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:57.348580 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:57.348528 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:57.348959 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:57.348689 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:34:58.348843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:58.348801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:34:58.349262 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:58.348856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:34:58.349262 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:58.348962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:34:58.349262 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:58.349085 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:34:59.349219 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:34:59.349184 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:34:59.349697 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:34:59.349303 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:00.348411 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:00.348370 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:00.348610 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:00.348374 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:00.348610 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:00.348530 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:00.348737 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:00.348614 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:00.907499 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:00.907451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:00.907986 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:00.907654 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:35:00.907986 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:00.907727 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:16.907708746 +0000 UTC m=+34.122873756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:35:01.007855 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:01.007812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:01.008036 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:01.007887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:01.008036 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.007981 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:35:01.008036 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.008011 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:35:01.008036 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.008025 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:35:01.008036 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.008036 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:35:01.008251 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.008046 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:17.008027209 +0000 UTC m=+34.223192230 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:35:01.008251 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.008070 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:17.008061259 +0000 UTC m=+34.223226269 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:35:01.351816 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:01.351788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:01.351953 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:01.351788 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:01.351953 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.351882 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:01.352031 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:01.351952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:02.349257 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:02.349224 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:02.349731 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:02.349352 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:03.349866 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:03.349837 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:03.350385 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:03.349935 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:03.350385 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:03.350014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:03.350385 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:03.350099 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:04.348979 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:04.348940 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:04.349167 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:04.349086 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:05.349188 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:05.349149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:05.349806 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:05.349292 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:05.349806 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:05.349698 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:05.349806 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:05.349797 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:06.348988 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.348636 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:06.349146 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:06.349077 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:06.481182 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.481148 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tr8pb" event={"ID":"703e1173-0804-4069-8845-99e75777cd3b","Type":"ContainerStarted","Data":"51ed2af67417eb1c0aba5d3b7238b9643f2c26f7121877367562f478283ad852"} Jul 29 18:35:06.483012 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.482983 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wpmjn" event={"ID":"5730df58-4eca-443a-8bbf-31ffe7667927","Type":"ContainerStarted","Data":"b9a67cd36a30683a8327229167815d877d0add69d6779a7e47237c395e93eede"} Jul 29 18:35:06.484609 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.484589 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wdk2w" event={"ID":"afd0e76f-9dd0-429b-95ad-214ee9962d1d","Type":"ContainerStarted","Data":"752094621c0232c59f9278c9d282836de4d55622522029cd6e672e0c8ef4dc30"} Jul 29 18:35:06.487384 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487358 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:35:06.487753 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487730 2578 generic.go:356] "Generic (PLEG): container finished" podID="e637449e-59bf-41ce-a871-63f3c81263fe" containerID="502e7acfe014b8e5196dbcd87a2020275d97327d0b2052293723322b815fd426" exitCode=1 Jul 29 18:35:06.487863 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487806 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"9345911046e5c9013cb85ea8273faab7191b48cf68b7226b40f8c27d28b24acd"} Jul 29 18:35:06.487936 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487873 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"bf55c22ecb1a597f70d5eee4767da8f2ae3c08b12d010ab95941f37531751139"} Jul 29 18:35:06.487936 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487911 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"f90a35de350d6c9972aa441f2398b84a6c064794abfbd2338c6757f2d646c62f"} Jul 29 18:35:06.488038 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.487945 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerDied","Data":"502e7acfe014b8e5196dbcd87a2020275d97327d0b2052293723322b815fd426"} Jul 29 18:35:06.488038 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.488008 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"adcd666308b61fa140c66ff820843bbdaa95cce6de4a4aa0399bcc1225ea08fd"} Jul 29 18:35:06.489414 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.489391 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="eebeae44954c82380157cb6fb423b7a520edb060322b6c84214cfb7c5e9665c5" exitCode=0 Jul 29 18:35:06.489555 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.489466 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"eebeae44954c82380157cb6fb423b7a520edb060322b6c84214cfb7c5e9665c5"} Jul 29 18:35:06.493746 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.493719 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbhmw" event={"ID":"33461b5b-deeb-4b99-af20-390ad7ccdac6","Type":"ContainerStarted","Data":"3e804d8266732ca421e21190ee1b1f4d6c9fca6c9e8e6bf685e6a3193cbb11b3"} Jul 29 18:35:06.495289 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.495266 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" event={"ID":"3d7ea662-72c5-4589-9b1a-0e04d9268ba3","Type":"ContainerStarted","Data":"ca5d8cb932da2fd9dd13fc85383927df9c2eb6a9df639911ba7fee7ea725480d"} Jul 29 18:35:06.496865 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.496823 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" event={"ID":"4fbc4845-56bc-4ff0-b6b6-c49acddbce00","Type":"ContainerStarted","Data":"8a4cb37ac1f6a4e598353ca67ce01adc4ec72096c761d45b7897161fc3c9afad"} Jul 29 18:35:06.500924 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.500887 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tr8pb" podStartSLOduration=3.432744048 podStartE2EDuration="22.50087549s" totalImagesPullingTime="19.068131442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:06.500428666 +0000 UTC m=+23.715593695" watchObservedRunningTime="2026-07-29 18:35:06.50087549 +0000 UTC m=+23.716040575" Jul 29 18:35:06.501030 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.500970 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-174.ec2.internal" podStartSLOduration=21.500961896 podStartE2EDuration="21.500961896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.459344395 +0000 UTC m=+5.674509428" watchObservedRunningTime="2026-07-29 18:35:06.500961896 +0000 UTC m=+23.716126927" Jul 29 18:35:06.525297 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.525245 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2xg8x" podStartSLOduration=3.443994297 podStartE2EDuration="22.525228936s" totalImagesPullingTime="19.081234639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:06.524412226 +0000 UTC m=+23.739577257" watchObservedRunningTime="2026-07-29 18:35:06.525228936 +0000 UTC m=+23.740393990" Jul 29 18:35:06.543633 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.543591 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wpmjn" podStartSLOduration=3.469667167 podStartE2EDuration="22.543577144s" totalImagesPullingTime="19.073909977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:06.543364585 +0000 UTC m=+23.758529615" watchObservedRunningTime="2026-07-29 18:35:06.543577144 +0000 UTC m=+23.758742174" Jul 29 18:35:06.559129 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.559086 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dbhmw" podStartSLOduration=3.488844965 podStartE2EDuration="22.559068308s" totalImagesPullingTime="19.070223343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:06.558708485 +0000 UTC m=+23.773873513" watchObservedRunningTime="2026-07-29 18:35:06.559068308 +0000 UTC m=+23.774233343" Jul 29 18:35:06.575431 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.575388 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wdk2w" podStartSLOduration=3.241364296 podStartE2EDuration="22.575374113s" totalImagesPullingTime="19.334009817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:06.574926082 +0000 UTC m=+23.790091111" watchObservedRunningTime="2026-07-29 18:35:06.575374113 +0000 UTC m=+23.790539141" Jul 29 18:35:06.966662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:06.966635 2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 18:35:07.224404 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.224194 2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T18:35:06.966657417Z","UUID":"05ffeeaa-f41d-4010-92b4-48b5e96e9e8c","Handler":null,"Name":"","Endpoint":""} Jul 29 18:35:07.226020 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.225995 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 18:35:07.226151 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.226042 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 18:35:07.349706 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.349675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:07.349893 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.349675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:07.349893 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:07.349808 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:07.350007 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:07.349896 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:07.501813 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.501687 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-qz6nh" event={"ID":"c9dd8bf2-7f5b-4d60-a20f-6c1251a90596","Type":"ContainerStarted","Data":"ff3030ac86ff02a23e042bf7ff9a636cea0f30f22fb4eb59588f8803f5eccfb7"} Jul 29 18:35:07.506024 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.505991 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:35:07.506562 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.506490 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"d4ab759844fc1dbbffde0a26c01af33e8bcad290513d9ebd4356b79bc5924c58"} Jul 29 18:35:07.509764 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.509729 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" event={"ID":"3d7ea662-72c5-4589-9b1a-0e04d9268ba3","Type":"ContainerStarted","Data":"7b9ff76453a3be1019028f402679b4d567bc41f6e9a279518ec3cf5bd79de57d"} Jul 29 18:35:07.528869 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:07.528807 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-qz6nh" podStartSLOduration=4.458153604 podStartE2EDuration="23.528731897s" totalImagesPullingTime="19.070578293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:07.527979867 +0000 UTC m=+24.743144897" watchObservedRunningTime="2026-07-29 18:35:07.528731897 +0000 UTC m=+24.743896920" Jul 29 18:35:08.348765 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:08.348733 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:08.348942 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:08.348867 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:08.514352 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:08.514310 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" event={"ID":"3d7ea662-72c5-4589-9b1a-0e04d9268ba3","Type":"ContainerStarted","Data":"60fbfc28a5c3632c5d55d1eb6d2f397d0d7ba8bc94fc12c66ac33f760431874b"} Jul 29 18:35:08.535490 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:08.535415 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4qvr" podStartSLOduration=7.284319879 podStartE2EDuration="24.535392548s" totalImagesPullingTime="17.251072669s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:08.534832244 +0000 UTC m=+25.749997304" watchObservedRunningTime="2026-07-29 18:35:08.535392548 +0000 UTC m=+25.750557578" Jul 29 18:35:09.348726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:09.348691 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:09.348939 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:09.348737 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:09.348939 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:09.348887 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:09.349057 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:09.348952 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:09.520708 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:09.520681 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:35:09.521157 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:09.521133 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"d10e19919050fa71cf23216aed6e0ddcb12ce65c76efc4cbdbbdb816949ab69a"} Jul 29 18:35:10.348418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:10.348386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:10.348613 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:10.348532 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:11.280866 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.280672 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:35:11.281525 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.281304 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:35:11.348677 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.348638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:11.348677 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.348668 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:11.348942 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:11.348757 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:11.348942 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:11.348904 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:11.528771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.528746 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:35:11.529125 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.529102 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"6ca7e52489a1c5911a171f3a9ee01982801fbd97b7a568bb23bdba369cfb1ec3"} Jul 29 18:35:11.529513 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.529483 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:11.529635 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.529568 2578 scope.go:117] "RemoveContainer" containerID="502e7acfe014b8e5196dbcd87a2020275d97327d0b2052293723322b815fd426" Jul 29 18:35:11.531360 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.531288 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="98f455c1ef3bba35d525188ba218cfc954770678e4e06f0566eaf10fdb4b2718" exitCode=0 Jul 29 18:35:11.531465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.531357 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"98f455c1ef3bba35d525188ba218cfc954770678e4e06f0566eaf10fdb4b2718"} Jul 29 18:35:11.531657 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.531639 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:35:11.532143 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.532120 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbhmw" Jul 29 18:35:11.548050 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:11.548029 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:12.348821 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.348518 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:12.349226 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:12.348955 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:12.538459 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.538431 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:35:12.538903 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.538863 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" event={"ID":"e637449e-59bf-41ce-a871-63f3c81263fe","Type":"ContainerStarted","Data":"9068b36ba1cadb6f4325c6e8209e89618b033398346788ff775c72e477efa2af"} Jul 29 18:35:12.539085 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.539070 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:12.539177 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.539119 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:12.541763 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.541716 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="03bbc4515868283364e2cbf5ad526d80c1a340d682741565ba3116e75cd6624e" exitCode=0 Jul 29 18:35:12.541902 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.541783 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"03bbc4515868283364e2cbf5ad526d80c1a340d682741565ba3116e75cd6624e"} Jul 29 18:35:12.557474 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.557444 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:12.568361 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.568309 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" podStartSLOduration=9.138161917 podStartE2EDuration="28.568294679s" totalImagesPullingTime="19.430132762s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:12.56821687 +0000 UTC m=+29.783381926" watchObservedRunningTime="2026-07-29 18:35:12.568294679 +0000 UTC m=+29.783459704" Jul 29 18:35:12.904355 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.904265 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vwxh"] Jul 29 18:35:12.904564 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.904418 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:12.904564 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:12.904521 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:12.905048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.905025 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxz79"] Jul 29 18:35:12.905146 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.905134 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:12.905244 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:12.905227 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:12.916218 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.916190 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4h9l"] Jul 29 18:35:12.916329 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:12.916300 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:12.916408 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:12.916391 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:14.348709 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:14.348448 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:14.349183 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:14.348500 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:14.349183 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:14.348815 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:14.349183 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:14.348918 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:14.551380 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:14.551348 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="534578f93c7230e2de6581a35f6f630084b6aecf5426f0ee11744ada81fd4b51" exitCode=0 Jul 29 18:35:14.551523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:14.551389 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"534578f93c7230e2de6581a35f6f630084b6aecf5426f0ee11744ada81fd4b51"} Jul 29 18:35:15.348492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:15.348460 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:15.348686 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:15.348613 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:15.505519 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:15.505492 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wpmjn_5730df58-4eca-443a-8bbf-31ffe7667927/dns-node-resolver/0.log" Jul 29 18:35:16.285312 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:16.285280 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tr8pb_703e1173-0804-4069-8845-99e75777cd3b/node-ca/0.log" Jul 29 18:35:16.348738 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:16.348705 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:16.348914 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:16.348753 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:16.348914 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:16.348850 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:16.349050 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:16.348993 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:16.930713 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:16.930662 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:16.931215 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:16.930849 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:35:16.931215 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:16.930926 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret podName:a610eee9-34c8-4d8f-82df-a0405f0c7ff3 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:48.930903628 +0000 UTC m=+66.146068651 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret") pod "global-pull-secret-syncer-2vwxh" (UID: "a610eee9-34c8-4d8f-82df-a0405f0c7ff3") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:35:17.031502 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:17.031474 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:17.031671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:17.031518 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:17.031671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031632 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:35:17.031671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031642 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:35:17.031671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031659 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:35:17.031671 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031668 2578 projected.go:196] Error preparing data for projected volume kube-api-access-znk2n for pod openshift-network-diagnostics/network-check-target-qxz79: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:35:17.031861 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031696 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs podName:28edf3c8-c9a6-4b2f-a826-7644923b3345 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:49.031679405 +0000 UTC m=+66.246844421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs") pod "network-metrics-daemon-q4h9l" (UID: "28edf3c8-c9a6-4b2f-a826-7644923b3345") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:35:17.031861 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.031713 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n podName:f5c497be-ed20-4345-bdc9-6d5ca5467531 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:49.03170485 +0000 UTC m=+66.246869860 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-znk2n" (UniqueName: "kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n") pod "network-check-target-qxz79" (UID: "f5c497be-ed20-4345-bdc9-6d5ca5467531") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:35:17.349178 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:17.349145 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:17.349399 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:17.349258 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2vwxh" podUID="a610eee9-34c8-4d8f-82df-a0405f0c7ff3" Jul 29 18:35:18.348204 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.348171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:18.348813 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.348171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:18.348813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.348287 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qxz79" podUID="f5c497be-ed20-4345-bdc9-6d5ca5467531" Jul 29 18:35:18.348813 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.348362 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-q4h9l" podUID="28edf3c8-c9a6-4b2f-a826-7644923b3345" Jul 29 18:35:18.551279 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.551249 2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-174.ec2.internal" event="NodeReady" Jul 29 18:35:18.551461 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.551401 2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 18:35:18.598671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.598638 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-p2hrm"] Jul 29 18:35:18.621576 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.621174 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-67sk6"] Jul 29 18:35:18.621576 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.621267 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.624822 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.624781 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-q2mp4\"" Jul 29 18:35:18.625141 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.625113 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 18:35:18.625249 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.625228 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 18:35:18.625479 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.625457 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 18:35:18.637396 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.637356 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p2hrm"] Jul 29 18:35:18.637396 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.637398 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.637724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.637451 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67sk6"] Jul 29 18:35:18.637724 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.637489 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ztqnv"] Jul 29 18:35:18.640872 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.640848 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 18:35:18.641043 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.641023 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 18:35:18.641108 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.641068 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8px9m\"" Jul 29 18:35:18.641445 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.641426 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 18:35:18.641579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.641453 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 18:35:18.661384 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.661329 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztqnv"] Jul 29 18:35:18.661645 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.661558 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.664100 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.664077 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 18:35:18.664331 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.664110 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hgmvm\"" Jul 29 18:35:18.664331 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.664287 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 18:35:18.742367 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742329 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77ec832-fc29-45f0-9ad7-19ce55fe5600-tmp-dir\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.742580 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742381 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgnqt\" (UniqueName: \"kubernetes.io/projected/c77ec832-fc29-45f0-9ad7-19ce55fe5600-kube-api-access-rgnqt\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.742580 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742454 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.742580 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ae797e-8c19-4639-b2b9-79d792c53009-cert\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.742580 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.742748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fssw\" (UniqueName: \"kubernetes.io/projected/04ae797e-8c19-4639-b2b9-79d792c53009-kube-api-access-2fssw\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.742748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5dw\" (UniqueName: \"kubernetes.io/projected/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-api-access-6f5dw\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.742748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.742748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e41bc761-fa0b-4f31-889f-154be250fbbb-crio-socket\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.742748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742700 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77ec832-fc29-45f0-9ad7-19ce55fe5600-config-volume\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.742914 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.742774 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e41bc761-fa0b-4f31-889f-154be250fbbb-data-volume\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.843859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.843762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fssw\" (UniqueName: \"kubernetes.io/projected/04ae797e-8c19-4639-b2b9-79d792c53009-kube-api-access-2fssw\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.843859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.843802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5dw\" (UniqueName: \"kubernetes.io/projected/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-api-access-6f5dw\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.843859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.843827 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.844085 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.843985 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 18:35:18.844408 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.844374 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:19.344277968 +0000 UTC m=+36.559442976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : secret "insights-runtime-extractor-tls" not found Jul 29 18:35:18.844518 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e41bc761-fa0b-4f31-889f-154be250fbbb-crio-socket\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.844518 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844501 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77ec832-fc29-45f0-9ad7-19ce55fe5600-config-volume\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.844667 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844605 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e41bc761-fa0b-4f31-889f-154be250fbbb-data-volume\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.844769 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844719 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e41bc761-fa0b-4f31-889f-154be250fbbb-crio-socket\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.844902 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77ec832-fc29-45f0-9ad7-19ce55fe5600-tmp-dir\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.844902 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844872 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgnqt\" (UniqueName: \"kubernetes.io/projected/c77ec832-fc29-45f0-9ad7-19ce55fe5600-kube-api-access-rgnqt\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.845014 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.844929 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.845077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.845048 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e41bc761-fa0b-4f31-889f-154be250fbbb-data-volume\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.845144 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.845126 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ae797e-8c19-4639-b2b9-79d792c53009-cert\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.845313 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.845295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.845411 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.845274 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c77ec832-fc29-45f0-9ad7-19ce55fe5600-tmp-dir\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.845465 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.845452 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 18:35:18.847607 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.845689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c77ec832-fc29-45f0-9ad7-19ce55fe5600-config-volume\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.847607 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.845946 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls podName:c77ec832-fc29-45f0-9ad7-19ce55fe5600 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:19.345928309 +0000 UTC m=+36.561093316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls") pod "dns-default-ztqnv" (UID: "c77ec832-fc29-45f0-9ad7-19ce55fe5600") : secret "dns-default-metrics-tls" not found Jul 29 18:35:18.847783 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.847649 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 18:35:18.847783 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:18.847739 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:19.347720728 +0000 UTC m=+36.562885751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : configmap "kube-rbac-proxy" not found Jul 29 18:35:18.850967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.850949 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/04ae797e-8c19-4639-b2b9-79d792c53009-cert\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.854118 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.854093 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgnqt\" (UniqueName: \"kubernetes.io/projected/c77ec832-fc29-45f0-9ad7-19ce55fe5600-kube-api-access-rgnqt\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:18.855169 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.855147 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5dw\" (UniqueName: \"kubernetes.io/projected/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-api-access-6f5dw\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:18.855256 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.855184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fssw\" (UniqueName: \"kubernetes.io/projected/04ae797e-8c19-4639-b2b9-79d792c53009-kube-api-access-2fssw\") pod \"ingress-canary-p2hrm\" (UID: \"04ae797e-8c19-4639-b2b9-79d792c53009\") " pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:18.932897 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:18.932846 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-p2hrm" Jul 29 18:35:19.323520 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.323280 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-p2hrm"] Jul 29 18:35:19.326980 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:19.326952 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ae797e_8c19_4639_b2b9_79d792c53009.slice/crio-50894e1b10ba1d7fe89c6d99df81ba466e8467b5b1f873aff1098fb799ec3568 WatchSource:0}: Error finding container 50894e1b10ba1d7fe89c6d99df81ba466e8467b5b1f873aff1098fb799ec3568: Status 404 returned error can't find the container with id 50894e1b10ba1d7fe89c6d99df81ba466e8467b5b1f873aff1098fb799ec3568 Jul 29 18:35:19.349187 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.349153 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:19.350665 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.350645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:19.350800 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.350750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:19.350862 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350813 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 18:35:19.350916 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350873 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 18:35:19.350916 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350877 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 18:35:19.350916 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.350790 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:19.350916 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350905 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:20.3508831 +0000 UTC m=+37.566048112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : secret "insights-runtime-extractor-tls" not found Jul 29 18:35:19.351064 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350933 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:20.350921571 +0000 UTC m=+37.566086592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : configmap "kube-rbac-proxy" not found Jul 29 18:35:19.351064 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:19.350953 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls podName:c77ec832-fc29-45f0-9ad7-19ce55fe5600 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:20.350943955 +0000 UTC m=+37.566108971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls") pod "dns-default-ztqnv" (UID: "c77ec832-fc29-45f0-9ad7-19ce55fe5600") : secret "dns-default-metrics-tls" not found Jul 29 18:35:19.351921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.351904 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:19.568352 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.568315 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p2hrm" event={"ID":"04ae797e-8c19-4639-b2b9-79d792c53009","Type":"ContainerStarted","Data":"50894e1b10ba1d7fe89c6d99df81ba466e8467b5b1f873aff1098fb799ec3568"} Jul 29 18:35:19.571067 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.571036 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="ea142b5644f20b61bd5948f32c582836e9c21e645acedb19cbd739ae370fb431" exitCode=0 Jul 29 18:35:19.571199 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:19.571077 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"ea142b5644f20b61bd5948f32c582836e9c21e645acedb19cbd739ae370fb431"} Jul 29 18:35:20.348596 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.348559 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:20.348769 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.348742 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:20.351368 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.351335 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:20.352504 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.352483 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b24qk\"" Jul 29 18:35:20.352645 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.352516 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:20.352645 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.352533 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7g2r\"" Jul 29 18:35:20.352645 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.352609 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:20.359129 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.359106 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:20.359241 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.359156 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:20.359241 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.359188 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:20.359241 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359213 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 18:35:20.359241 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359231 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 18:35:20.359431 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359273 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls podName:c77ec832-fc29-45f0-9ad7-19ce55fe5600 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:22.359253095 +0000 UTC m=+39.574418110 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls") pod "dns-default-ztqnv" (UID: "c77ec832-fc29-45f0-9ad7-19ce55fe5600") : secret "dns-default-metrics-tls" not found Jul 29 18:35:20.359431 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359291 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:22.35928162 +0000 UTC m=+39.574446630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : configmap "kube-rbac-proxy" not found Jul 29 18:35:20.359431 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359298 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 18:35:20.359431 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:20.359353 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:22.359337365 +0000 UTC m=+39.574502377 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : secret "insights-runtime-extractor-tls" not found Jul 29 18:35:20.577713 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.577677 2578 generic.go:356] "Generic (PLEG): container finished" podID="cf0d1aa8-2165-473b-a77d-25e2af93723a" containerID="4b9ccf110bcfd4bc9f22ff76e92177649f3592b8123ce025a61e5568d02fe601" exitCode=0 Jul 29 18:35:20.577864 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:20.577748 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerDied","Data":"4b9ccf110bcfd4bc9f22ff76e92177649f3592b8123ce025a61e5568d02fe601"} Jul 29 18:35:21.584942 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:21.584902 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t68xd" event={"ID":"cf0d1aa8-2165-473b-a77d-25e2af93723a","Type":"ContainerStarted","Data":"3b0e1dc4e846a2eea40ae58412dd7f144296e337322488974d30780b6be2b31f"} Jul 29 18:35:21.609959 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:21.609168 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t68xd" podStartSLOduration=11.386033209 podStartE2EDuration="37.60914492s" totalImagesPullingTime="26.223111711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:21.607185954 +0000 UTC m=+38.822350985" watchObservedRunningTime="2026-07-29 18:35:21.60914492 +0000 UTC m=+38.824309950" Jul 29 18:35:22.207739 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.207705 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-q777v"] Jul 29 18:35:22.222734 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.222651 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-q777v"] Jul 29 18:35:22.222865 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.222792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.225641 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.225611 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 29 18:35:22.225641 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.225632 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 29 18:35:22.225859 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.225722 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 29 18:35:22.226014 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.225983 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zp4ls\"" Jul 29 18:35:22.226584 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.226567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 29 18:35:22.374885 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.374841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd7t6\" (UniqueName: \"kubernetes.io/projected/0613de46-c083-4b42-97cd-e603935a9ff7-kube-api-access-wd7t6\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.375057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.374901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:22.375057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.374970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:22.375057 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375011 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 18:35:22.375057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.375018 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0613de46-c083-4b42-97cd-e603935a9ff7-signing-cabundle\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.375057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.375038 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375062 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls podName:c77ec832-fc29-45f0-9ad7-19ce55fe5600 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.375047007 +0000 UTC m=+43.590212017 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls") pod "dns-default-ztqnv" (UID: "c77ec832-fc29-45f0-9ad7-19ce55fe5600") : secret "dns-default-metrics-tls" not found Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.375080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0613de46-c083-4b42-97cd-e603935a9ff7-signing-key\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375114 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375115 2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375154 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.375143092 +0000 UTC m=+43.590308099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : secret "insights-runtime-extractor-tls" not found Jul 29 18:35:22.375277 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:22.375179 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.375158986 +0000 UTC m=+43.590323996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : configmap "kube-rbac-proxy" not found Jul 29 18:35:22.475780 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.475700 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0613de46-c083-4b42-97cd-e603935a9ff7-signing-cabundle\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.475780 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.475744 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0613de46-c083-4b42-97cd-e603935a9ff7-signing-key\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.475955 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.475822 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd7t6\" (UniqueName: \"kubernetes.io/projected/0613de46-c083-4b42-97cd-e603935a9ff7-kube-api-access-wd7t6\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.476477 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.476458 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/0613de46-c083-4b42-97cd-e603935a9ff7-signing-cabundle\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.480289 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.480267 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/0613de46-c083-4b42-97cd-e603935a9ff7-signing-key\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.485202 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.485162 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd7t6\" (UniqueName: \"kubernetes.io/projected/0613de46-c083-4b42-97cd-e603935a9ff7-kube-api-access-wd7t6\") pod \"service-ca-658d7b7c78-q777v\" (UID: \"0613de46-c083-4b42-97cd-e603935a9ff7\") " pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.532931 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.532898 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-q777v" Jul 29 18:35:22.590567 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.590510 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-p2hrm" event={"ID":"04ae797e-8c19-4639-b2b9-79d792c53009","Type":"ContainerStarted","Data":"58dd14b408a6a2900f467ac51d30b26c4a4c6219a8ec2e7859b7ada2b5b53a8d"} Jul 29 18:35:22.616577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.616148 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-p2hrm" podStartSLOduration=1.981470219 podStartE2EDuration="4.616117069s" totalImagesPullingTime="2.63464685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:22.614833065 +0000 UTC m=+39.829998098" watchObservedRunningTime="2026-07-29 18:35:22.616117069 +0000 UTC m=+39.831282098" Jul 29 18:35:22.686392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:22.686357 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-q777v"] Jul 29 18:35:22.702605 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:22.702570 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0613de46_c083_4b42_97cd_e603935a9ff7.slice/crio-6346ade33ce5b6a2d7316ad70caa4332b9e6d2ec205010196393f93b819f64eb WatchSource:0}: Error finding container 6346ade33ce5b6a2d7316ad70caa4332b9e6d2ec205010196393f93b819f64eb: Status 404 returned error can't find the container with id 6346ade33ce5b6a2d7316ad70caa4332b9e6d2ec205010196393f93b819f64eb Jul 29 18:35:23.595299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:23.595023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-q777v" event={"ID":"0613de46-c083-4b42-97cd-e603935a9ff7","Type":"ContainerStarted","Data":"6346ade33ce5b6a2d7316ad70caa4332b9e6d2ec205010196393f93b819f64eb"} Jul 29 18:35:25.602180 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:25.602145 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-q777v" event={"ID":"0613de46-c083-4b42-97cd-e603935a9ff7","Type":"ContainerStarted","Data":"87284cab79a01081551b0050f01413eaf0d35c7942f27d22bb0ca2ada02d7a47"} Jul 29 18:35:25.621171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:25.621119 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-q777v" podStartSLOduration=1.293547166 podStartE2EDuration="3.621102279s" totalImagesPullingTime="2.327555113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:25.620472723 +0000 UTC m=+42.835637752" watchObservedRunningTime="2026-07-29 18:35:25.621102279 +0000 UTC m=+42.836267308" Jul 29 18:35:26.407921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:26.407880 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:26.407921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:26.407932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:26.408204 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:26.407955 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:26.408204 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:26.408057 2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 18:35:26.408204 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:26.408065 2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 18:35:26.408204 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:26.408117 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls podName:e41bc761-fa0b-4f31-889f-154be250fbbb nodeName:}" failed. No retries permitted until 2026-07-29 18:35:34.408100487 +0000 UTC m=+51.623265502 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-67sk6" (UID: "e41bc761-fa0b-4f31-889f-154be250fbbb") : secret "insights-runtime-extractor-tls" not found Jul 29 18:35:26.408204 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:26.408131 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls podName:c77ec832-fc29-45f0-9ad7-19ce55fe5600 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:34.4081256 +0000 UTC m=+51.623290611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls") pod "dns-default-ztqnv" (UID: "c77ec832-fc29-45f0-9ad7-19ce55fe5600") : secret "dns-default-metrics-tls" not found Jul 29 18:35:26.408458 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:26.408440 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e41bc761-fa0b-4f31-889f-154be250fbbb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:34.472322 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.472277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:34.472827 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.472333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:34.475434 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.475400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e41bc761-fa0b-4f31-889f-154be250fbbb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-67sk6\" (UID: \"e41bc761-fa0b-4f31-889f-154be250fbbb\") " pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:34.475579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.475434 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c77ec832-fc29-45f0-9ad7-19ce55fe5600-metrics-tls\") pod \"dns-default-ztqnv\" (UID: \"c77ec832-fc29-45f0-9ad7-19ce55fe5600\") " pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:34.549193 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.549152 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-67sk6" Jul 29 18:35:34.571811 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.571787 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:34.733251 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.733135 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-67sk6"] Jul 29 18:35:34.736805 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:34.736769 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode41bc761_fa0b_4f31_889f_154be250fbbb.slice/crio-1d7316c241085dedf9d2222aa4d0fdeeed39512e3b0d705f3a49441612dfe95e WatchSource:0}: Error finding container 1d7316c241085dedf9d2222aa4d0fdeeed39512e3b0d705f3a49441612dfe95e: Status 404 returned error can't find the container with id 1d7316c241085dedf9d2222aa4d0fdeeed39512e3b0d705f3a49441612dfe95e Jul 29 18:35:34.763245 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:34.763209 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ztqnv"] Jul 29 18:35:34.766612 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:34.766587 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77ec832_fc29_45f0_9ad7_19ce55fe5600.slice/crio-abf70cc0a9c6ab59ab7c22b4479be3d808b280f66c57132d639bc66414b44e73 WatchSource:0}: Error finding container abf70cc0a9c6ab59ab7c22b4479be3d808b280f66c57132d639bc66414b44e73: Status 404 returned error can't find the container with id abf70cc0a9c6ab59ab7c22b4479be3d808b280f66c57132d639bc66414b44e73 Jul 29 18:35:35.629585 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:35.629517 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztqnv" event={"ID":"c77ec832-fc29-45f0-9ad7-19ce55fe5600","Type":"ContainerStarted","Data":"abf70cc0a9c6ab59ab7c22b4479be3d808b280f66c57132d639bc66414b44e73"} Jul 29 18:35:35.631250 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:35.631218 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67sk6" event={"ID":"e41bc761-fa0b-4f31-889f-154be250fbbb","Type":"ContainerStarted","Data":"1d7316c241085dedf9d2222aa4d0fdeeed39512e3b0d705f3a49441612dfe95e"} Jul 29 18:35:40.651744 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:40.651694 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztqnv" event={"ID":"c77ec832-fc29-45f0-9ad7-19ce55fe5600","Type":"ContainerStarted","Data":"df39df997366329fb94db5005d2b282f925ea7a02b8353f5e6a8145dbc8fa080"} Jul 29 18:35:40.652228 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:40.651766 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ztqnv" event={"ID":"c77ec832-fc29-45f0-9ad7-19ce55fe5600","Type":"ContainerStarted","Data":"0470fdb19bb83aa8c23fcc50a4b45a0ad7c54a42426ec4c90d2db70a1a5fc1fb"} Jul 29 18:35:40.652228 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:40.651825 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:40.657397 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:40.657358 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67sk6" event={"ID":"e41bc761-fa0b-4f31-889f-154be250fbbb","Type":"ContainerStarted","Data":"8253ef3bf16cf91d37194024ffb44a4525e425439ed17335ba6fabe2fe529b86"} Jul 29 18:35:40.677047 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:40.676983 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ztqnv" podStartSLOduration=17.436566186 podStartE2EDuration="22.676959385s" totalImagesPullingTime="5.240393199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.674883965 +0000 UTC m=+57.890048995" watchObservedRunningTime="2026-07-29 18:35:40.676959385 +0000 UTC m=+57.892124414" Jul 29 18:35:44.295734 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.295697 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p"] Jul 29 18:35:44.297467 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.297449 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.298979 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.298954 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn"] Jul 29 18:35:44.300743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.300719 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 18:35:44.300841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.300780 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 18:35:44.301537 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.301523 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.301855 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.301834 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 18:35:44.301966 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.301949 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 29 18:35:44.305672 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.305655 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7456j\"" Jul 29 18:35:44.306986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.306972 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 29 18:35:44.316080 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.316062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p"] Jul 29 18:35:44.317132 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.317112 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn"] Jul 29 18:35:44.347031 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.346996 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4571b20-9823-447b-aa00-74ff8b9f0817-klusterlet-config\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.347193 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.347044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g46xp\" (UniqueName: \"kubernetes.io/projected/a7a84a4f-b868-423b-920b-eccff3db3475-kube-api-access-g46xp\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.347193 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.347091 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7a84a4f-b868-423b-920b-eccff3db3475-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.347193 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.347111 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4571b20-9823-447b-aa00-74ff8b9f0817-tmp\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.347193 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.347132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz85z\" (UniqueName: \"kubernetes.io/projected/d4571b20-9823-447b-aa00-74ff8b9f0817-kube-api-access-tz85z\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.388826 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.388793 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-94cdb7f8d-gqpx8"] Jul 29 18:35:44.390668 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.390647 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.393385 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.393365 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 18:35:44.393692 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.393672 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 18:35:44.393806 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.393696 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-v2s6m\"" Jul 29 18:35:44.393850 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.393840 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 18:35:44.398364 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.398343 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 18:35:44.412937 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.412909 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94cdb7f8d-gqpx8"] Jul 29 18:35:44.447459 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz85z\" (UniqueName: \"kubernetes.io/projected/d4571b20-9823-447b-aa00-74ff8b9f0817-kube-api-access-tz85z\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.447459 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447459 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-image-registry-private-configuration\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447481 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-installation-pull-secrets\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447511 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-certificates\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-trusted-ca\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4571b20-9823-447b-aa00-74ff8b9f0817-klusterlet-config\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447638 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7a84a4f-b868-423b-920b-eccff3db3475-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447667 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4571b20-9823-447b-aa00-74ff8b9f0817-tmp\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.447742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447708 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plpmd\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-kube-api-access-plpmd\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.448077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447735 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g46xp\" (UniqueName: \"kubernetes.io/projected/a7a84a4f-b868-423b-920b-eccff3db3475-kube-api-access-g46xp\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.448077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-tls\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.448077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447848 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-ca-trust-extracted\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.448077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.447880 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-bound-sa-token\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.448077 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.448060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4571b20-9823-447b-aa00-74ff8b9f0817-tmp\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.450172 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.450151 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/d4571b20-9823-447b-aa00-74ff8b9f0817-klusterlet-config\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.450290 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.450254 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a7a84a4f-b868-423b-920b-eccff3db3475-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.466926 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.466897 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g46xp\" (UniqueName: \"kubernetes.io/projected/a7a84a4f-b868-423b-920b-eccff3db3475-kube-api-access-g46xp\") pod \"managed-serviceaccount-addon-agent-869dcd58d8-759pn\" (UID: \"a7a84a4f-b868-423b-920b-eccff3db3475\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.467049 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.466954 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz85z\" (UniqueName: \"kubernetes.io/projected/d4571b20-9823-447b-aa00-74ff8b9f0817-kube-api-access-tz85z\") pod \"klusterlet-addon-workmgr-c5d9556bf-p8d7p\" (UID: \"d4571b20-9823-447b-aa00-74ff8b9f0817\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.549048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.548952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-image-registry-private-configuration\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.548993 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-installation-pull-secrets\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549016 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-certificates\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549042 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-trusted-ca\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plpmd\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-kube-api-access-plpmd\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549172 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-tls\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549192 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-ca-trust-extracted\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-bound-sa-token\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.549764 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.549732 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-ca-trust-extracted\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.550045 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.550026 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-certificates\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.550358 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.550336 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-trusted-ca\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.551811 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.551792 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-image-registry-private-configuration\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.551876 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.551837 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-registry-tls\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.551876 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.551861 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-installation-pull-secrets\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.562415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.562392 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jrhmw" Jul 29 18:35:44.564242 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.564211 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-bound-sa-token\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.564353 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.564339 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plpmd\" (UniqueName: \"kubernetes.io/projected/ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed-kube-api-access-plpmd\") pod \"image-registry-94cdb7f8d-gqpx8\" (UID: \"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed\") " pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.608454 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.608151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:44.625100 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.625062 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" Jul 29 18:35:44.674980 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.674862 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67sk6" event={"ID":"e41bc761-fa0b-4f31-889f-154be250fbbb","Type":"ContainerStarted","Data":"1f5fb67517948c9f5d3bb5d42143345f1561782019d1f1225446f9fbb139e12a"} Jul 29 18:35:44.699638 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.699451 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:44.751113 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.751060 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p"] Jul 29 18:35:44.754885 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:44.754854 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4571b20_9823_447b_aa00_74ff8b9f0817.slice/crio-a500a26d33168b8b674f58530998eee54b26da0d895ac2bd2e7fa564cb37441c WatchSource:0}: Error finding container a500a26d33168b8b674f58530998eee54b26da0d895ac2bd2e7fa564cb37441c: Status 404 returned error can't find the container with id a500a26d33168b8b674f58530998eee54b26da0d895ac2bd2e7fa564cb37441c Jul 29 18:35:44.764268 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.764237 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn"] Jul 29 18:35:44.768046 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:44.768011 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7a84a4f_b868_423b_920b_eccff3db3475.slice/crio-fd61212d88021b9ed6b195bb071589b592f716e43170b1361eabaad5ec55a079 WatchSource:0}: Error finding container fd61212d88021b9ed6b195bb071589b592f716e43170b1361eabaad5ec55a079: Status 404 returned error can't find the container with id fd61212d88021b9ed6b195bb071589b592f716e43170b1361eabaad5ec55a079 Jul 29 18:35:44.854529 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:44.854427 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-94cdb7f8d-gqpx8"] Jul 29 18:35:44.857984 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:44.857955 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff8efbe7_3e36_4ecb_a9e3_b518740ca2ed.slice/crio-2dec5e98bcba1b256f28faef0c063cbfd6b7c37f7d8cf163a82c4a7a5bb48a90 WatchSource:0}: Error finding container 2dec5e98bcba1b256f28faef0c063cbfd6b7c37f7d8cf163a82c4a7a5bb48a90: Status 404 returned error can't find the container with id 2dec5e98bcba1b256f28faef0c063cbfd6b7c37f7d8cf163a82c4a7a5bb48a90 Jul 29 18:35:45.683426 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.683392 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" event={"ID":"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed","Type":"ContainerStarted","Data":"5271f5d4580b96667e1dcc790f984f1b5cab9c059831668ac6db127ad07c9da5"} Jul 29 18:35:45.683906 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.683452 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" event={"ID":"ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed","Type":"ContainerStarted","Data":"2dec5e98bcba1b256f28faef0c063cbfd6b7c37f7d8cf163a82c4a7a5bb48a90"} Jul 29 18:35:45.684302 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.684281 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:35:45.686087 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.686019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" event={"ID":"a7a84a4f-b868-423b-920b-eccff3db3475","Type":"ContainerStarted","Data":"fd61212d88021b9ed6b195bb071589b592f716e43170b1361eabaad5ec55a079"} Jul 29 18:35:45.688372 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.688301 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" event={"ID":"d4571b20-9823-447b-aa00-74ff8b9f0817","Type":"ContainerStarted","Data":"a500a26d33168b8b674f58530998eee54b26da0d895ac2bd2e7fa564cb37441c"} Jul 29 18:35:45.704855 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:45.704492 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" podStartSLOduration=1.704474603 podStartE2EDuration="1.704474603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:45.703447453 +0000 UTC m=+62.918612482" watchObservedRunningTime="2026-07-29 18:35:45.704474603 +0000 UTC m=+62.919639629" Jul 29 18:35:48.994062 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:48.994025 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:48.996743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:48.996720 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:49.007525 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.007494 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a610eee9-34c8-4d8f-82df-a0405f0c7ff3-original-pull-secret\") pod \"global-pull-secret-syncer-2vwxh\" (UID: \"a610eee9-34c8-4d8f-82df-a0405f0c7ff3\") " pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:49.058702 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.058662 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2vwxh" Jul 29 18:35:49.095352 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.095315 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:49.095536 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.095373 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:49.098295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.098275 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:49.099349 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.099323 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:49.109394 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.109241 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:49.109394 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.109325 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28edf3c8-c9a6-4b2f-a826-7644923b3345-metrics-certs\") pod \"network-metrics-daemon-q4h9l\" (UID: \"28edf3c8-c9a6-4b2f-a826-7644923b3345\") " pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:49.120214 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.120192 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znk2n\" (UniqueName: \"kubernetes.io/projected/f5c497be-ed20-4345-bdc9-6d5ca5467531-kube-api-access-znk2n\") pod \"network-check-target-qxz79\" (UID: \"f5c497be-ed20-4345-bdc9-6d5ca5467531\") " pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:49.162734 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.162684 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g7g2r\"" Jul 29 18:35:49.168441 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.168415 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b24qk\"" Jul 29 18:35:49.170381 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.170360 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-q4h9l" Jul 29 18:35:49.176587 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.176560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:49.570208 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.570171 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:35:49.584406 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.584350 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.585588 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.585539 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:35:49.587368 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.587005 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:35:49.587368 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.587184 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:35:49.587564 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.587387 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:35:49.587619 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.587566 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:35:49.588367 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.588236 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:35:49.588464 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.588440 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m998h\"" Jul 29 18:35:49.592015 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.591993 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:35:49.598987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.598963 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:35:49.699792 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699760 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.699986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699807 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.699986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26746\" (UniqueName: \"kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.699986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.699986 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699971 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.700176 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.699998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.800940 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.800898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.800951 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.800980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26746\" (UniqueName: \"kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801033 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801055 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801076 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801812 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801762 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801812 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801798 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.801812 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.801809 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.804192 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.804168 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.804330 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.804313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.814810 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.814787 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26746\" (UniqueName: \"kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746\") pod \"console-6d9f8f5fff-wt7tz\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:49.897460 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:49.897375 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:50.665105 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:50.665072 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ztqnv" Jul 29 18:35:51.217995 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.217912 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2vwxh"] Jul 29 18:35:51.231772 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.231661 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:35:51.254291 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.254242 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qxz79"] Jul 29 18:35:51.257709 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.257620 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-q4h9l"] Jul 29 18:35:51.257932 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:51.257906 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c497be_ed20_4345_bdc9_6d5ca5467531.slice/crio-d8054a0621194c501fadb753a89c2e0d5c1a9371fa1ebbc565ab58f85dab50e9 WatchSource:0}: Error finding container d8054a0621194c501fadb753a89c2e0d5c1a9371fa1ebbc565ab58f85dab50e9: Status 404 returned error can't find the container with id d8054a0621194c501fadb753a89c2e0d5c1a9371fa1ebbc565ab58f85dab50e9 Jul 29 18:35:51.260827 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:51.260790 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28edf3c8_c9a6_4b2f_a826_7644923b3345.slice/crio-69212f909ae64b61e3b1a81096764e679b9cf3e264981c7ebb17a24ed082206d WatchSource:0}: Error finding container 69212f909ae64b61e3b1a81096764e679b9cf3e264981c7ebb17a24ed082206d: Status 404 returned error can't find the container with id 69212f909ae64b61e3b1a81096764e679b9cf3e264981c7ebb17a24ed082206d Jul 29 18:35:51.713867 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.713827 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxz79" event={"ID":"f5c497be-ed20-4345-bdc9-6d5ca5467531","Type":"ContainerStarted","Data":"d8054a0621194c501fadb753a89c2e0d5c1a9371fa1ebbc565ab58f85dab50e9"} Jul 29 18:35:51.721664 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.721576 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-67sk6" event={"ID":"e41bc761-fa0b-4f31-889f-154be250fbbb","Type":"ContainerStarted","Data":"e867708b6ae77dcfeed10c4f7795d55faf119b05a518a50439b9248d0519f815"} Jul 29 18:35:51.723222 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.723152 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4h9l" event={"ID":"28edf3c8-c9a6-4b2f-a826-7644923b3345","Type":"ContainerStarted","Data":"69212f909ae64b61e3b1a81096764e679b9cf3e264981c7ebb17a24ed082206d"} Jul 29 18:35:51.726219 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.726134 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f8f5fff-wt7tz" event={"ID":"9fa0eb56-54b1-48b8-ad6a-696a4763c622","Type":"ContainerStarted","Data":"96959737c12a9380a026959b0828abdbe00eaa4465decab0f050064adffd62c6"} Jul 29 18:35:51.732403 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.732325 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vwxh" event={"ID":"a610eee9-34c8-4d8f-82df-a0405f0c7ff3","Type":"ContainerStarted","Data":"b8cfccde8bd6a276f0c83337c1f4208c4a5c4dc40d116afd59665c49fe9f0b76"} Jul 29 18:35:51.735525 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.734930 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" event={"ID":"a7a84a4f-b868-423b-920b-eccff3db3475","Type":"ContainerStarted","Data":"f31ab882651bb22d2bc70e6d86e3904ad0058b963cd28690541562b6e36cd4d6"} Jul 29 18:35:51.739205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.738250 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-67sk6" podStartSLOduration=17.562382032 podStartE2EDuration="33.738236808s" totalImagesPullingTime="16.175854776s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:51.737606756 +0000 UTC m=+68.952771787" watchObservedRunningTime="2026-07-29 18:35:51.738236808 +0000 UTC m=+68.953401836" Jul 29 18:35:51.739205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.738751 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" event={"ID":"d4571b20-9823-447b-aa00-74ff8b9f0817","Type":"ContainerStarted","Data":"24470af6327747f98dbf6f87cea1ef29ed48968e7774f8ca1d4d699acb98609b"} Jul 29 18:35:51.739205 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.739095 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:51.744078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.744057 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" Jul 29 18:35:51.753266 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.753224 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-869dcd58d8-759pn" podStartSLOduration=1.501447482 podStartE2EDuration="7.753210539s" totalImagesPullingTime="6.251763057s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:51.752826554 +0000 UTC m=+68.967991582" watchObservedRunningTime="2026-07-29 18:35:51.753210539 +0000 UTC m=+68.968375567" Jul 29 18:35:51.775579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:51.774449 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-c5d9556bf-p8d7p" podStartSLOduration=1.487712714 podStartE2EDuration="7.774427125s" totalImagesPullingTime="6.286714411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:51.774024325 +0000 UTC m=+68.989189354" watchObservedRunningTime="2026-07-29 18:35:51.774427125 +0000 UTC m=+68.989592158" Jul 29 18:35:53.754852 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:53.754816 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4h9l" event={"ID":"28edf3c8-c9a6-4b2f-a826-7644923b3345","Type":"ContainerStarted","Data":"be8293071daae141570e7f4bc52f2bc86c8f8771a9e2e7fb6da368fcea30ca59"} Jul 29 18:35:53.755322 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:53.754877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-q4h9l" event={"ID":"28edf3c8-c9a6-4b2f-a826-7644923b3345","Type":"ContainerStarted","Data":"d1d36fc9482f6dec7be859c912fd0598e89dcdf2c30ad974c76eeac52c113667"} Jul 29 18:35:53.771940 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:53.771878 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-q4h9l" podStartSLOduration=68.418587287 podStartE2EDuration="1m9.771856553s" totalImagesPullingTime="1.353269266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:53.771299499 +0000 UTC m=+70.986464529" watchObservedRunningTime="2026-07-29 18:35:53.771856553 +0000 UTC m=+70.987021582" Jul 29 18:35:55.352443 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.352416 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:35:55.356105 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.356068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.363905 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.363862 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:35:55.366299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.366275 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:35:55.454778 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454739 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.454967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454794 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.454967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454829 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.454967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.454967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454888 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.454967 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.454920 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d95r9\" (UniqueName: \"kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.455188 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.455034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555623 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d95r9\" (UniqueName: \"kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555837 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555837 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555963 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555865 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555963 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555903 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555963 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555927 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.555963 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.555956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.556569 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.556490 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.556848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.556825 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.556975 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.556947 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.557249 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.557228 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.559083 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.559060 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.559185 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.559102 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.564175 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.564155 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d95r9\" (UniqueName: \"kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9\") pod \"console-6c646cbf5c-6768x\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:55.670482 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:55.670389 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:35:56.704365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.704308 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm"] Jul 29 18:35:56.733347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.733081 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-24z2z"] Jul 29 18:35:56.734650 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.733949 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.736579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.736368 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 18:35:56.736579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.736386 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 18:35:56.736909 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.736889 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 18:35:56.737847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.737532 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 18:35:56.737847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.737567 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 18:35:56.738272 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.738064 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 18:35:56.738272 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.738140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-9f9gk\"" Jul 29 18:35:56.748296 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.748247 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm"] Jul 29 18:35:56.748393 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.748379 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.751400 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.751384 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 18:35:56.751729 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.751707 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 18:35:56.751960 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.751944 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 18:35:56.752179 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.752162 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2fkkq\"" Jul 29 18:35:56.765935 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.765910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.766052 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.765953 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s26lp\" (UniqueName: \"kubernetes.io/projected/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-api-access-s26lp\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.766052 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.765979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.766052 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.766044 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.766178 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.766092 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9357e329-51f2-4227-ac96-e73e3e81a0d6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.766178 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.766123 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.866845 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866810 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-wtmp\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-tls\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866898 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866957 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-sys\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.866991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s26lp\" (UniqueName: \"kubernetes.io/projected/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-api-access-s26lp\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867020 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867075 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-textfile\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867175 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzcj\" (UniqueName: \"kubernetes.io/projected/38f8b54c-882f-4036-be37-7f42cdbc0f5d-kube-api-access-gtzcj\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-metrics-client-ca\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867264 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9357e329-51f2-4227-ac96-e73e3e81a0d6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867339 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867330 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.867705 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867360 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-root\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.867705 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:56.867539 2578 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 29 18:35:56.867705 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:56.867643 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls podName:9357e329-51f2-4227-ac96-e73e3e81a0d6 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:57.367622391 +0000 UTC m=+74.582787409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-lbnbm" (UID: "9357e329-51f2-4227-ac96-e73e3e81a0d6") : secret "kube-state-metrics-tls" not found Jul 29 18:35:56.867858 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.867835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.868096 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.868074 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9357e329-51f2-4227-ac96-e73e3e81a0d6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.868381 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.868361 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9357e329-51f2-4227-ac96-e73e3e81a0d6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.870247 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.870222 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.878396 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.878375 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s26lp\" (UniqueName: \"kubernetes.io/projected/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-api-access-s26lp\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:56.967901 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.967812 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-textfile\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.967901 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.967887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzcj\" (UniqueName: \"kubernetes.io/projected/38f8b54c-882f-4036-be37-7f42cdbc0f5d-kube-api-access-gtzcj\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.967928 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-metrics-client-ca\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.967952 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.967983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-root\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-wtmp\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968045 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-tls\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968067 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968097 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-sys\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968131 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-root\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968184 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-textfile\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968189 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-sys\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968313 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-wtmp\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968642 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968518 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-metrics-client-ca\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.968642 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.968531 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-accelerators-collector-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.971085 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.971061 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-tls\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.977586 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.977562 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzcj\" (UniqueName: \"kubernetes.io/projected/38f8b54c-882f-4036-be37-7f42cdbc0f5d-kube-api-access-gtzcj\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:56.977947 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:56.977915 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38f8b54c-882f-4036-be37-7f42cdbc0f5d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-24z2z\" (UID: \"38f8b54c-882f-4036-be37-7f42cdbc0f5d\") " pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:57.059713 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.059672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-24z2z" Jul 29 18:35:57.262910 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:57.262712 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38f8b54c_882f_4036_be37_7f42cdbc0f5d.slice/crio-e8af1d3c5fe9677e5ac9e9b2b5187a579a5c2c084bff22971e4f49330105948b WatchSource:0}: Error finding container e8af1d3c5fe9677e5ac9e9b2b5187a579a5c2c084bff22971e4f49330105948b: Status 404 returned error can't find the container with id e8af1d3c5fe9677e5ac9e9b2b5187a579a5c2c084bff22971e4f49330105948b Jul 29 18:35:57.370714 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.370678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:57.373387 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.373332 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9357e329-51f2-4227-ac96-e73e3e81a0d6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lbnbm\" (UID: \"9357e329-51f2-4227-ac96-e73e3e81a0d6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:57.416493 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.415968 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:35:57.420072 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:57.420048 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceb0f44_00a6_4508_8415_cbe513796b0c.slice/crio-9627ac6f993f29281b2989b259929d44065c8e209c61e5cafb79892ef9b51b3b WatchSource:0}: Error finding container 9627ac6f993f29281b2989b259929d44065c8e209c61e5cafb79892ef9b51b3b: Status 404 returned error can't find the container with id 9627ac6f993f29281b2989b259929d44065c8e209c61e5cafb79892ef9b51b3b Jul 29 18:35:57.646924 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.646894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" Jul 29 18:35:57.777649 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.775749 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:57.781510 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.779664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.784001 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.784224 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.784414 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.784703 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.784868 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.785028 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.785175 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.786311 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f8lrd\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.786694 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 18:35:57.787254 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.786891 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 18:35:57.803396 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.803344 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c646cbf5c-6768x" event={"ID":"dceb0f44-00a6-4508-8415-cbe513796b0c","Type":"ContainerStarted","Data":"23be1095c4b0c118fc96a28e3271e764e0c0031eca62b76a2ffe30245a0d997a"} Jul 29 18:35:57.803576 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.803399 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c646cbf5c-6768x" event={"ID":"dceb0f44-00a6-4508-8415-cbe513796b0c","Type":"ContainerStarted","Data":"9627ac6f993f29281b2989b259929d44065c8e209c61e5cafb79892ef9b51b3b"} Jul 29 18:35:57.805393 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.805277 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qxz79" event={"ID":"f5c497be-ed20-4345-bdc9-6d5ca5467531","Type":"ContainerStarted","Data":"f12f3f141095efcf38920d63af85fc620f34a571ce0dcd6cf3372f751758aec0"} Jul 29 18:35:57.805393 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.805356 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:35:57.809536 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.809512 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f8f5fff-wt7tz" event={"ID":"9fa0eb56-54b1-48b8-ad6a-696a4763c622","Type":"ContainerStarted","Data":"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47"} Jul 29 18:35:57.812593 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.812131 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2vwxh" event={"ID":"a610eee9-34c8-4d8f-82df-a0405f0c7ff3","Type":"ContainerStarted","Data":"06a5c894b8919c13dc572f50b55075daa1ff042e555ef54655d8654ae78c7d94"} Jul 29 18:35:57.814252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.814232 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24z2z" event={"ID":"38f8b54c-882f-4036-be37-7f42cdbc0f5d","Type":"ContainerStarted","Data":"e8af1d3c5fe9677e5ac9e9b2b5187a579a5c2c084bff22971e4f49330105948b"} Jul 29 18:35:57.825949 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.825902 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qxz79" podStartSLOduration=67.822183985 podStartE2EDuration="1m13.825887735s" totalImagesPullingTime="6.00370375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:57.8246687 +0000 UTC m=+75.039833760" watchObservedRunningTime="2026-07-29 18:35:57.825887735 +0000 UTC m=+75.041052765" Jul 29 18:35:57.831970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.831905 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm"] Jul 29 18:35:57.834739 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:57.834714 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9357e329_51f2_4227_ac96_e73e3e81a0d6.slice/crio-584a6a921df7c6df6aeb5e85b71bc7542ebf745cdb706d083e2e75dc34a62d36 WatchSource:0}: Error finding container 584a6a921df7c6df6aeb5e85b71bc7542ebf745cdb706d083e2e75dc34a62d36: Status 404 returned error can't find the container with id 584a6a921df7c6df6aeb5e85b71bc7542ebf745cdb706d083e2e75dc34a62d36 Jul 29 18:35:57.849110 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.849082 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:57.849980 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.849932 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c646cbf5c-6768x" podStartSLOduration=2.849914423 podStartE2EDuration="2.849914423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:57.847529209 +0000 UTC m=+75.062694237" watchObservedRunningTime="2026-07-29 18:35:57.849914423 +0000 UTC m=+75.065079454" Jul 29 18:35:57.863004 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.862944 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-2vwxh" podStartSLOduration=67.816964381 podStartE2EDuration="1m13.862926933s" totalImagesPullingTime="6.045962552s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:57.862525564 +0000 UTC m=+75.077690594" watchObservedRunningTime="2026-07-29 18:35:57.862926933 +0000 UTC m=+75.078091958" Jul 29 18:35:57.876677 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.876658 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.876789 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.876688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.876789 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.876707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.876789 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.876750 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.876948 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.876887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877031 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877082 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877046 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877139 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877194 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877159 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877357 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877409 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877371 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqgxm\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877409 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877402 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.877503 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.877484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.978777 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978679 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.978777 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.978777 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978771 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978805 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978836 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:57.978876 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle podName:88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:58.478851248 +0000 UTC m=+75.694016257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0") : configmap references non-existent config key: ca-bundle.crt Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978954 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqgxm\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.978976 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979037 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.979001 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979382 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.979064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979382 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.979096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979382 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.979118 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.979382 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.979160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.980778 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.980269 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.980778 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:57.980375 2578 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 29 18:35:57.980778 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:35:57.980428 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls podName:88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:58.480411281 +0000 UTC m=+75.695576287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0") : secret "alertmanager-main-tls" not found Jul 29 18:35:57.981013 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.980867 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.981903 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.981876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.983842 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.983805 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.984054 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.984032 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.984132 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.984062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.984186 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.984158 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.984413 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.984366 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.985857 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.985833 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.986190 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.986124 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:57.987787 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:57.987767 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqgxm\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.484348 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.484247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.484348 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.484341 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.485341 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.485310 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.487261 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.487235 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.702742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.702707 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:58.820886 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.820846 2578 generic.go:356] "Generic (PLEG): container finished" podID="38f8b54c-882f-4036-be37-7f42cdbc0f5d" containerID="8ff96c08a2bedaa75fef2e734c644f3c0c78586fd2d911c9ac88e0c9b742c411" exitCode=0 Jul 29 18:35:58.821252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.821169 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24z2z" event={"ID":"38f8b54c-882f-4036-be37-7f42cdbc0f5d","Type":"ContainerDied","Data":"8ff96c08a2bedaa75fef2e734c644f3c0c78586fd2d911c9ac88e0c9b742c411"} Jul 29 18:35:58.823538 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.823514 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" event={"ID":"9357e329-51f2-4227-ac96-e73e3e81a0d6","Type":"ContainerStarted","Data":"584a6a921df7c6df6aeb5e85b71bc7542ebf745cdb706d083e2e75dc34a62d36"} Jul 29 18:35:58.841890 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.841836 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6d9f8f5fff-wt7tz" podStartSLOduration=3.822319667 podStartE2EDuration="9.841819569s" totalImagesPullingTime="6.019499902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:57.880697562 +0000 UTC m=+75.095862593" watchObservedRunningTime="2026-07-29 18:35:58.841819569 +0000 UTC m=+76.056984595" Jul 29 18:35:58.862218 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:58.862183 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:59.042853 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:35:59.042819 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a405bd_3ebb_412f_a9bc_c6c5b8f44ed0.slice/crio-bdbe7df93c178f32d86e297c298b897c6584e2d47179a8894c5521f93f7047a0 WatchSource:0}: Error finding container bdbe7df93c178f32d86e297c298b897c6584e2d47179a8894c5521f93f7047a0: Status 404 returned error can't find the container with id bdbe7df93c178f32d86e297c298b897c6584e2d47179a8894c5521f93f7047a0 Jul 29 18:35:59.831603 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.831538 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24z2z" event={"ID":"38f8b54c-882f-4036-be37-7f42cdbc0f5d","Type":"ContainerStarted","Data":"5bf42065b9afe772324c24ed7963b7e018d57f42ce8e6d325e30230cfc03736c"} Jul 29 18:35:59.832078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.831637 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-24z2z" event={"ID":"38f8b54c-882f-4036-be37-7f42cdbc0f5d","Type":"ContainerStarted","Data":"3f0f3bc67a71a2c14b990f08a7f10a095046d2c55311975b018252a902fba7ad"} Jul 29 18:35:59.833965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.833936 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" event={"ID":"9357e329-51f2-4227-ac96-e73e3e81a0d6","Type":"ContainerStarted","Data":"4982d1335e30324f88288749d79a2bea088452da9aa0d1b110413eb9b6f519d3"} Jul 29 18:35:59.834093 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.834001 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" event={"ID":"9357e329-51f2-4227-ac96-e73e3e81a0d6","Type":"ContainerStarted","Data":"bed89f1fd1266b8efb65c3e762145fcd95a5233fae03038c239d9ba4dc93f8b4"} Jul 29 18:35:59.834093 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.834029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" event={"ID":"9357e329-51f2-4227-ac96-e73e3e81a0d6","Type":"ContainerStarted","Data":"9de71e9987c6d67368070742a0328550a99800c0c4d8b98c4306e538df8393db"} Jul 29 18:35:59.835337 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.835311 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"bdbe7df93c178f32d86e297c298b897c6584e2d47179a8894c5521f93f7047a0"} Jul 29 18:35:59.855083 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.855023 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-24z2z" podStartSLOduration=2.879388172 podStartE2EDuration="3.855004524s" totalImagesPullingTime="902.316602ms" totalInitContainerRuntime="73.29975ms" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:59.854467584 +0000 UTC m=+77.069632615" watchObservedRunningTime="2026-07-29 18:35:59.855004524 +0000 UTC m=+77.070169552" Jul 29 18:35:59.878211 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.878152 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lbnbm" podStartSLOduration=2.645240608 podStartE2EDuration="3.878129071s" totalImagesPullingTime="1.232888463s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:59.876044004 +0000 UTC m=+77.091209033" watchObservedRunningTime="2026-07-29 18:35:59.878129071 +0000 UTC m=+77.093294101" Jul 29 18:35:59.897611 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.897584 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:59.897746 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.897717 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:35:59.902961 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:35:59.902942 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:36:00.843231 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:00.843192 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a" exitCode=0 Jul 29 18:36:00.843729 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:00.843289 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a"} Jul 29 18:36:00.847494 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:00.847475 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:36:02.558453 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.558375 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:36:02.854857 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.854777 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99"} Jul 29 18:36:02.854857 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.854851 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238"} Jul 29 18:36:02.855025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.854877 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7"} Jul 29 18:36:02.855025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.854907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4"} Jul 29 18:36:02.855025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:02.854932 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c"} Jul 29 18:36:03.011756 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.011722 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:03.019153 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.019121 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.022403 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.022381 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 18:36:03.029035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.029003 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 18:36:03.029035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.029024 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 18:36:03.030362 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.030242 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 18:36:03.030362 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.030268 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 18:36:03.030815 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.030458 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 18:36:03.031429 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.031328 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5agf8kiql0qsk\"" Jul 29 18:36:03.031496 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.031432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 18:36:03.031584 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.031539 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w9vrs\"" Jul 29 18:36:03.031664 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.031641 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 18:36:03.031790 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.031773 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 18:36:03.032248 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.032121 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 18:36:03.032248 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.032177 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 18:36:03.032248 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.032230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 18:36:03.032450 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.032261 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 18:36:03.033666 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.033644 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 18:36:03.035916 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.035897 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 18:36:03.036291 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.036269 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:03.127758 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127758 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127762 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127818 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127855 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127893 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127910 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127924 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.127952 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.127938 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9c6q\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128034 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128058 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128076 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128152 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128185 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128246 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128241 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128509 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128266 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128509 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128292 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.128509 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.128331 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.228861 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228831 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.228861 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228868 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228887 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228902 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228921 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.228943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229119 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229098 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229145 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229184 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229212 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229238 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9c6q\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229292 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229356 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229318 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229370 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229395 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229403 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229515 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229598 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.229752 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229654 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.230323 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.229955 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.230323 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.230296 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.230435 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.230381 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.232590 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.232281 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.232590 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.232307 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.233435 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.233405 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.233899 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.233873 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.233997 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.233922 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.234097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.234073 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.234163 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.234097 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.234432 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.234409 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.235403 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.235378 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.235501 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.235473 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.236128 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.236099 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.236700 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.236679 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.236894 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.236871 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.237156 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.237138 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.237461 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.237439 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.239858 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.239834 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9c6q\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q\") pod \"prometheus-k8s-0\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.335940 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.335894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:03.487919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.487887 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:03.491505 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:36:03.491476 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9289b45_f6f1_4eff_b322_9336d7b1573f.slice/crio-f42d3a2f621de2c409752a7b23f8bfc127e0e03671532c62131fb339d45d8c10 WatchSource:0}: Error finding container f42d3a2f621de2c409752a7b23f8bfc127e0e03671532c62131fb339d45d8c10: Status 404 returned error can't find the container with id f42d3a2f621de2c409752a7b23f8bfc127e0e03671532c62131fb339d45d8c10 Jul 29 18:36:03.862232 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.862188 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerStarted","Data":"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46"} Jul 29 18:36:03.863805 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.863778 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" exitCode=0 Jul 29 18:36:03.863909 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.863833 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} Jul 29 18:36:03.863909 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.863878 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"f42d3a2f621de2c409752a7b23f8bfc127e0e03671532c62131fb339d45d8c10"} Jul 29 18:36:03.892094 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:03.892034 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.029105118 podStartE2EDuration="6.892015251s" totalImagesPullingTime="2.84524135s" totalInitContainerRuntime="17.668783ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:03.889468236 +0000 UTC m=+81.104633299" watchObservedRunningTime="2026-07-29 18:36:03.892015251 +0000 UTC m=+81.107180272" Jul 29 18:36:05.671013 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:05.670978 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:36:06.883929 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:06.883882 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} Jul 29 18:36:06.884355 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:06.884023 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} Jul 29 18:36:07.699781 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:07.699752 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-94cdb7f8d-gqpx8" Jul 29 18:36:08.896294 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:08.896255 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} Jul 29 18:36:08.896753 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:08.896324 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} Jul 29 18:36:09.905047 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:09.905015 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} Jul 29 18:36:09.905468 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:09.905078 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerStarted","Data":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} Jul 29 18:36:09.935347 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:09.935284 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.218316121 podStartE2EDuration="7.935263272s" totalImagesPullingTime="4.700541589s" totalInitContainerRuntime="16.405562ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:36:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:09.932596081 +0000 UTC m=+87.147761109" watchObservedRunningTime="2026-07-29 18:36:09.935263272 +0000 UTC m=+87.150428300" Jul 29 18:36:13.336007 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:13.335976 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:23.288734 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:23.288691 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:36:27.578202 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:27.578160 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c646cbf5c-6768x" podUID="dceb0f44-00a6-4508-8415-cbe513796b0c" containerName="console" containerID="cri-o://23be1095c4b0c118fc96a28e3271e764e0c0031eca62b76a2ffe30245a0d997a" gracePeriod=15 Jul 29 18:36:27.982029 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:27.981995 2578 generic.go:356] "Generic (PLEG): container finished" podID="dceb0f44-00a6-4508-8415-cbe513796b0c" containerID="23be1095c4b0c118fc96a28e3271e764e0c0031eca62b76a2ffe30245a0d997a" exitCode=0 Jul 29 18:36:27.982173 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:27.982070 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c646cbf5c-6768x" event={"ID":"dceb0f44-00a6-4508-8415-cbe513796b0c","Type":"ContainerDied","Data":"23be1095c4b0c118fc96a28e3271e764e0c0031eca62b76a2ffe30245a0d997a"} Jul 29 18:36:28.342800 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.342758 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:36:28.350886 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.350847 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.350920 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.350957 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d95r9\" (UniqueName: \"kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351001 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351023 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351046 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351078 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351077 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle\") pod \"dceb0f44-00a6-4508-8415-cbe513796b0c\" (UID: \"dceb0f44-00a6-4508-8415-cbe513796b0c\") " Jul 29 18:36:28.351396 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351299 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config" (OuterVolumeSpecName: "console-config") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:28.351520 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351496 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca" (OuterVolumeSpecName: "service-ca") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:28.351673 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351641 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:28.351743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.351713 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:28.353648 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.353603 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:28.353766 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.353683 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9" (OuterVolumeSpecName: "kube-api-access-d95r9") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "kube-api-access-d95r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:28.353766 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.353697 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "dceb0f44-00a6-4508-8415-cbe513796b0c" (UID: "dceb0f44-00a6-4508-8415-cbe513796b0c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:28.452519 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452480 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-oauth-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452519 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452512 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d95r9\" (UniqueName: \"kubernetes.io/projected/dceb0f44-00a6-4508-8415-cbe513796b0c-kube-api-access-d95r9\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452519 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452521 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-service-ca\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452519 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452530 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-oauth-serving-cert\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452566 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/dceb0f44-00a6-4508-8415-cbe513796b0c-console-serving-cert\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452577 2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-trusted-ca-bundle\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.452847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.452587 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/dceb0f44-00a6-4508-8415-cbe513796b0c-console-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:28.826940 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.826906 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qxz79" Jul 29 18:36:28.987577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.987538 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c646cbf5c-6768x" Jul 29 18:36:28.987745 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.987535 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c646cbf5c-6768x" event={"ID":"dceb0f44-00a6-4508-8415-cbe513796b0c","Type":"ContainerDied","Data":"9627ac6f993f29281b2989b259929d44065c8e209c61e5cafb79892ef9b51b3b"} Jul 29 18:36:28.987745 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:28.987693 2578 scope.go:117] "RemoveContainer" containerID="23be1095c4b0c118fc96a28e3271e764e0c0031eca62b76a2ffe30245a0d997a" Jul 29 18:36:29.013051 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:29.013026 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:36:29.016840 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:29.016818 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c646cbf5c-6768x"] Jul 29 18:36:29.352957 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:29.352925 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dceb0f44-00a6-4508-8415-cbe513796b0c" path="/var/lib/kubelet/pods/dceb0f44-00a6-4508-8415-cbe513796b0c/volumes" Jul 29 18:36:48.308702 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.308646 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6d9f8f5fff-wt7tz" podUID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" containerName="console" containerID="cri-o://e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47" gracePeriod=15 Jul 29 18:36:48.555503 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.555479 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:36:48.636934 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.636846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.636934 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.636884 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.636934 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.636921 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.637232 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.636944 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26746\" (UniqueName: \"kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.637232 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.637009 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.637232 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.637042 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert\") pod \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\" (UID: \"9fa0eb56-54b1-48b8-ad6a-696a4763c622\") " Jul 29 18:36:48.637403 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.637340 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:48.637486 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.637458 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config" (OuterVolumeSpecName: "console-config") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:48.637613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.637590 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:48.639415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.639388 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746" (OuterVolumeSpecName: "kube-api-access-26746") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "kube-api-access-26746". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:48.639415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.639391 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:48.639596 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.639453 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fa0eb56-54b1-48b8-ad6a-696a4763c622" (UID: "9fa0eb56-54b1-48b8-ad6a-696a4763c622"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:48.738613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738581 2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:48.738613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738608 2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-oauth-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:48.738613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738618 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26746\" (UniqueName: \"kubernetes.io/projected/9fa0eb56-54b1-48b8-ad6a-696a4763c622-kube-api-access-26746\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:48.738863 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738627 2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fa0eb56-54b1-48b8-ad6a-696a4763c622-console-serving-cert\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:48.738863 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738639 2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-oauth-serving-cert\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:48.738863 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:48.738648 2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fa0eb56-54b1-48b8-ad6a-696a4763c622-service-ca\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:36:49.066355 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.066320 2578 generic.go:356] "Generic (PLEG): container finished" podID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" containerID="e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47" exitCode=0 Jul 29 18:36:49.066529 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.066366 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d9f8f5fff-wt7tz" Jul 29 18:36:49.066529 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.066408 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f8f5fff-wt7tz" event={"ID":"9fa0eb56-54b1-48b8-ad6a-696a4763c622","Type":"ContainerDied","Data":"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47"} Jul 29 18:36:49.066529 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.066457 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d9f8f5fff-wt7tz" event={"ID":"9fa0eb56-54b1-48b8-ad6a-696a4763c622","Type":"ContainerDied","Data":"96959737c12a9380a026959b0828abdbe00eaa4465decab0f050064adffd62c6"} Jul 29 18:36:49.066529 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.066478 2578 scope.go:117] "RemoveContainer" containerID="e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47" Jul 29 18:36:49.075281 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.075266 2578 scope.go:117] "RemoveContainer" containerID="e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47" Jul 29 18:36:49.075597 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:36:49.075576 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47\": container with ID starting with e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47 not found: ID does not exist" containerID="e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47" Jul 29 18:36:49.075640 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.075605 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47"} err="failed to get container status \"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47\": rpc error: code = NotFound desc = could not find container \"e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47\": container with ID starting with e892ce931f126580dbdf11413535927d816643decdf72d63f6da1b31c238ef47 not found: ID does not exist" Jul 29 18:36:49.088255 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.088231 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:36:49.092837 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.092814 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d9f8f5fff-wt7tz"] Jul 29 18:36:49.353135 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:36:49.353049 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" path="/var/lib/kubelet/pods/9fa0eb56-54b1-48b8-ad6a-696a4763c622/volumes" Jul 29 18:37:03.336527 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:03.336483 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:03.357043 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:03.357015 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:04.136852 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:04.136822 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:17.077905 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.077869 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:37:17.078315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078179 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="alertmanager" containerID="cri-o://9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" gracePeriod=120 Jul 29 18:37:17.078315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078269 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-web" containerID="cri-o://f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" gracePeriod=120 Jul 29 18:37:17.078315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078255 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-metric" containerID="cri-o://38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" gracePeriod=120 Jul 29 18:37:17.078487 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078318 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy" containerID="cri-o://1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" gracePeriod=120 Jul 29 18:37:17.078487 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078377 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="prom-label-proxy" containerID="cri-o://c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" gracePeriod=120 Jul 29 18:37:17.078487 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:17.078271 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="config-reloader" containerID="cri-o://2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" gracePeriod=120 Jul 29 18:37:18.181075 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181040 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" exitCode=0 Jul 29 18:37:18.181075 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181067 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" exitCode=0 Jul 29 18:37:18.181075 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181074 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" exitCode=0 Jul 29 18:37:18.181075 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181082 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" exitCode=0 Jul 29 18:37:18.181574 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46"} Jul 29 18:37:18.181574 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181170 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238"} Jul 29 18:37:18.181574 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181199 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4"} Jul 29 18:37:18.181574 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.181224 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c"} Jul 29 18:37:18.325918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.325893 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:37:18.396843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396757 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.396843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396794 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.396843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396830 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396862 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396887 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396922 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396946 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.396978 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397005 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wqgxm\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397044 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397120 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397071 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397512 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397131 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.397512 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397199 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") pod \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\" (UID: \"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0\") " Jul 29 18:37:18.398418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397953 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:18.398418 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.397967 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:18.399011 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.398972 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:37:18.400439 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.400410 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume" (OuterVolumeSpecName: "config-volume") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.400702 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.400676 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:37:18.401138 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.401111 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.401370 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.401346 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.402002 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.401971 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm" (OuterVolumeSpecName: "kube-api-access-wqgxm") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "kube-api-access-wqgxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:37:18.402202 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.402177 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.402297 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.402269 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out" (OuterVolumeSpecName: "config-out") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:37:18.402350 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.402287 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.405021 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.404982 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.412510 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.412480 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config" (OuterVolumeSpecName: "web-config") pod "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" (UID: "88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:18.498772 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498736 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498772 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498766 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498772 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498777 2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-cluster-tls-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498786 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498795 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498806 2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-volume\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498814 2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-secret-alertmanager-main-tls\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498822 2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-metrics-client-ca\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498831 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-web-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498839 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-config-out\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498846 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqgxm\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-kube-api-access-wqgxm\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498854 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-tls-assets\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:18.498987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:18.498861 2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0-alertmanager-main-db\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:19.193515 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193459 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" exitCode=0 Jul 29 18:37:19.193515 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193502 2578 generic.go:356] "Generic (PLEG): container finished" podID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerID="f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" exitCode=0 Jul 29 18:37:19.194071 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193561 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99"} Jul 29 18:37:19.194071 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193653 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:37:19.194071 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7"} Jul 29 18:37:19.194071 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193742 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0","Type":"ContainerDied","Data":"bdbe7df93c178f32d86e297c298b897c6584e2d47179a8894c5521f93f7047a0"} Jul 29 18:37:19.194071 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.193761 2578 scope.go:117] "RemoveContainer" containerID="c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" Jul 29 18:37:19.202235 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.202170 2578 scope.go:117] "RemoveContainer" containerID="38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" Jul 29 18:37:19.209624 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.209604 2578 scope.go:117] "RemoveContainer" containerID="1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" Jul 29 18:37:19.218410 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.218367 2578 scope.go:117] "RemoveContainer" containerID="f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" Jul 29 18:37:19.218835 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.218789 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:37:19.223964 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.223932 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:37:19.227662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.227636 2578 scope.go:117] "RemoveContainer" containerID="2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" Jul 29 18:37:19.236107 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.236087 2578 scope.go:117] "RemoveContainer" containerID="9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" Jul 29 18:37:19.244605 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.244584 2578 scope.go:117] "RemoveContainer" containerID="b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a" Jul 29 18:37:19.251927 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.251907 2578 scope.go:117] "RemoveContainer" containerID="c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" Jul 29 18:37:19.252254 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.252229 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46\": container with ID starting with c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46 not found: ID does not exist" containerID="c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" Jul 29 18:37:19.252337 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252262 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46"} err="failed to get container status \"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46\": rpc error: code = NotFound desc = could not find container \"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46\": container with ID starting with c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46 not found: ID does not exist" Jul 29 18:37:19.252337 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252283 2578 scope.go:117] "RemoveContainer" containerID="38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" Jul 29 18:37:19.252516 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.252497 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99\": container with ID starting with 38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99 not found: ID does not exist" containerID="38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" Jul 29 18:37:19.252617 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252523 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99"} err="failed to get container status \"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99\": rpc error: code = NotFound desc = could not find container \"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99\": container with ID starting with 38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99 not found: ID does not exist" Jul 29 18:37:19.252678 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252633 2578 scope.go:117] "RemoveContainer" containerID="1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" Jul 29 18:37:19.252883 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.252865 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238\": container with ID starting with 1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238 not found: ID does not exist" containerID="1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" Jul 29 18:37:19.252927 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252887 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238"} err="failed to get container status \"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238\": rpc error: code = NotFound desc = could not find container \"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238\": container with ID starting with 1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238 not found: ID does not exist" Jul 29 18:37:19.252927 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.252902 2578 scope.go:117] "RemoveContainer" containerID="f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" Jul 29 18:37:19.253121 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.253106 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7\": container with ID starting with f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7 not found: ID does not exist" containerID="f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" Jul 29 18:37:19.253157 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253126 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7"} err="failed to get container status \"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7\": rpc error: code = NotFound desc = could not find container \"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7\": container with ID starting with f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7 not found: ID does not exist" Jul 29 18:37:19.253157 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253139 2578 scope.go:117] "RemoveContainer" containerID="2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" Jul 29 18:37:19.253342 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.253328 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4\": container with ID starting with 2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4 not found: ID does not exist" containerID="2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" Jul 29 18:37:19.253385 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253346 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4"} err="failed to get container status \"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4\": rpc error: code = NotFound desc = could not find container \"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4\": container with ID starting with 2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4 not found: ID does not exist" Jul 29 18:37:19.253385 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253358 2578 scope.go:117] "RemoveContainer" containerID="9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" Jul 29 18:37:19.253602 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.253554 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c\": container with ID starting with 9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c not found: ID does not exist" containerID="9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" Jul 29 18:37:19.253602 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253574 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c"} err="failed to get container status \"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c\": rpc error: code = NotFound desc = could not find container \"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c\": container with ID starting with 9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c not found: ID does not exist" Jul 29 18:37:19.253602 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253587 2578 scope.go:117] "RemoveContainer" containerID="b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a" Jul 29 18:37:19.253793 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:19.253779 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a\": container with ID starting with b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a not found: ID does not exist" containerID="b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a" Jul 29 18:37:19.253833 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253798 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a"} err="failed to get container status \"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a\": rpc error: code = NotFound desc = could not find container \"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a\": container with ID starting with b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a not found: ID does not exist" Jul 29 18:37:19.253833 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253810 2578 scope.go:117] "RemoveContainer" containerID="c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46" Jul 29 18:37:19.254002 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.253987 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46"} err="failed to get container status \"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46\": rpc error: code = NotFound desc = could not find container \"c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46\": container with ID starting with c57998b22ae2ca4bd06f036053553e1e7614d7338fc61ce85c16c9dd3d7a2f46 not found: ID does not exist" Jul 29 18:37:19.254043 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254002 2578 scope.go:117] "RemoveContainer" containerID="38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99" Jul 29 18:37:19.254251 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254231 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99"} err="failed to get container status \"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99\": rpc error: code = NotFound desc = could not find container \"38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99\": container with ID starting with 38d130ecf818d317ea921cd56b5870473ad1032b4011f93f7493a4c7bcfbdf99 not found: ID does not exist" Jul 29 18:37:19.254308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254251 2578 scope.go:117] "RemoveContainer" containerID="1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238" Jul 29 18:37:19.254465 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254449 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238"} err="failed to get container status \"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238\": rpc error: code = NotFound desc = could not find container \"1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238\": container with ID starting with 1036f30c4f041e997f9c93597481376237a0afd6771319480365182532e0d238 not found: ID does not exist" Jul 29 18:37:19.254506 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254465 2578 scope.go:117] "RemoveContainer" containerID="f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7" Jul 29 18:37:19.254693 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254676 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7"} err="failed to get container status \"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7\": rpc error: code = NotFound desc = could not find container \"f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7\": container with ID starting with f66a478e521155bb0d5a6e2e6b394d28b7aaffbf2da8f0b55d3bda17f0cde0c7 not found: ID does not exist" Jul 29 18:37:19.254743 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254695 2578 scope.go:117] "RemoveContainer" containerID="2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4" Jul 29 18:37:19.254894 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254874 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4"} err="failed to get container status \"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4\": rpc error: code = NotFound desc = could not find container \"2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4\": container with ID starting with 2e81a8f1fd9ae5985f69dcd378e6c86d83978bdded083e2e95a236b0867c2ec4 not found: ID does not exist" Jul 29 18:37:19.254995 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.254896 2578 scope.go:117] "RemoveContainer" containerID="9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c" Jul 29 18:37:19.255143 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.255105 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c"} err="failed to get container status \"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c\": rpc error: code = NotFound desc = could not find container \"9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c\": container with ID starting with 9796bd7f6e9a948b0ea21ffff0dc39e49bd45cac0e28e182204dbea6e96e362c not found: ID does not exist" Jul 29 18:37:19.255143 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.255137 2578 scope.go:117] "RemoveContainer" containerID="b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a" Jul 29 18:37:19.255838 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.255812 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a"} err="failed to get container status \"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a\": rpc error: code = NotFound desc = could not find container \"b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a\": container with ID starting with b1a32a942865a6e89aca5f0616feb07344f4a6dbab6e3e0bae34d904f73d4a9a not found: ID does not exist" Jul 29 18:37:19.353807 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:19.353772 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" path="/var/lib/kubelet/pods/88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0/volumes" Jul 29 18:37:21.416305 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416268 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:21.416748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416629 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="prometheus" containerID="cri-o://8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" gracePeriod=600 Jul 29 18:37:21.416748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416649 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" gracePeriod=600 Jul 29 18:37:21.416748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416649 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="config-reloader" containerID="cri-o://17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" gracePeriod=600 Jul 29 18:37:21.416748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416629 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-web" containerID="cri-o://e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" gracePeriod=600 Jul 29 18:37:21.416748 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416729 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy" containerID="cri-o://c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" gracePeriod=600 Jul 29 18:37:21.416989 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.416773 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="thanos-sidecar" containerID="cri-o://cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" gracePeriod=600 Jul 29 18:37:21.660438 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.660410 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:21.724848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724750 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.724848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724792 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.724848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724821 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.724848 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724846 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724862 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724884 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724913 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724950 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.724980 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725218 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.725415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725306 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.725415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725314 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725415 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725332 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.725872 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725820 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.725872 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725868 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726006 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725913 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9c6q\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726006 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725946 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726006 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.725977 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726008 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726035 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726059 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726097 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726126 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets\") pod \"a9289b45-f6f1-4eff-b322-9336d7b1573f\" (UID: \"a9289b45-f6f1-4eff-b322-9336d7b1573f\") " Jul 29 18:37:21.726365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726054 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.726365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726321 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.726365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726351 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.726493 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726368 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.726493 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726402 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-metrics-client-ca\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.726834 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.726807 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:37:21.727704 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.727671 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.727813 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.727721 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:21.728149 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.728106 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.728329 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.728298 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config" (OuterVolumeSpecName: "config") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.730392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.730363 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.730494 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.730437 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.730804 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.730538 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q" (OuterVolumeSpecName: "kube-api-access-j9c6q") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "kube-api-access-j9c6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:37:21.730804 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.730569 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.731115 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.731087 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:37:21.731218 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.731140 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out" (OuterVolumeSpecName: "config-out") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:37:21.731335 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.731298 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.731406 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.731373 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.732269 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.732242 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.740168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.740137 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config" (OuterVolumeSpecName: "web-config") pod "a9289b45-f6f1-4eff-b322-9336d7b1573f" (UID: "a9289b45-f6f1-4eff-b322-9336d7b1573f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:21.826977 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.826921 2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.826977 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.826966 2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.826977 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.826977 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.826977 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.826987 2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-kube-rbac-proxy\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827008 2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-web-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827018 2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-tls-assets\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827025 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827035 2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-config\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827043 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827051 2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-grpc-tls\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827059 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-db\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827069 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827077 2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-metrics-client-certs\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827086 2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a9289b45-f6f1-4eff-b322-9336d7b1573f-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827093 2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a9289b45-f6f1-4eff-b322-9336d7b1573f-config-out\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827101 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9c6q\" (UniqueName: \"kubernetes.io/projected/a9289b45-f6f1-4eff-b322-9336d7b1573f-kube-api-access-j9c6q\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:21.827252 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:21.827109 2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9289b45-f6f1-4eff-b322-9336d7b1573f-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:37:22.212583 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212523 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" exitCode=0 Jul 29 18:37:22.212583 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212572 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" exitCode=0 Jul 29 18:37:22.212583 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212582 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" exitCode=0 Jul 29 18:37:22.212583 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212589 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" exitCode=0 Jul 29 18:37:22.212583 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212593 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" exitCode=0 Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212604 2578 generic.go:356] "Generic (PLEG): container finished" podID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" exitCode=0 Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212635 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212585 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212728 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212756 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212782 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212792 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212807 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212831 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} Jul 29 18:37:22.212970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.212855 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a9289b45-f6f1-4eff-b322-9336d7b1573f","Type":"ContainerDied","Data":"f42d3a2f621de2c409752a7b23f8bfc127e0e03671532c62131fb339d45d8c10"} Jul 29 18:37:22.222377 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.222356 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.230019 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.229990 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.237175 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.237154 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.241406 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.241383 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:22.244783 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.244767 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.255431 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.252105 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:22.255431 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.254188 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.262790 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.262760 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.270923 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.270902 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.271242 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.271221 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.271287 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271252 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.271287 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271271 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.271502 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.271485 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.271571 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271505 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.271571 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271517 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.271732 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.271717 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.271770 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271736 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.271770 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271748 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.271979 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.271959 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.272048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.271988 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.272048 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272012 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.272233 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.272219 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.272272 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272237 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.272272 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272249 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.272431 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.272416 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.272467 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272433 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.272467 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272444 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.272924 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:37:22.272909 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.272972 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272926 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.272972 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.272938 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.273341 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.273305 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.273430 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.273335 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.273920 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.273892 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.274016 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.273920 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.274313 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274192 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.274399 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274315 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.274653 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274614 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.274653 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274644 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.274911 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274894 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.274964 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.274920 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.275182 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275166 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.275231 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275187 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.275231 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275170 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:22.275473 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275453 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.275473 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275473 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275494 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="alertmanager" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275506 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="alertmanager" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275515 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275521 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275527 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" containerName="console" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275532 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" containerName="console" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275564 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="config-reloader" Jul 29 18:37:22.275577 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275574 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275586 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-metric" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275594 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-metric" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275603 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="prometheus" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275611 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="prometheus" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275621 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275628 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275637 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="init-config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275646 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="init-config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275663 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dceb0f44-00a6-4508-8415-cbe513796b0c" containerName="console" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275670 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="dceb0f44-00a6-4508-8415-cbe513796b0c" containerName="console" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275680 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="prom-label-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275688 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="prom-label-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275696 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275705 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275714 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275720 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275727 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="thanos-sidecar" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275732 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="thanos-sidecar" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275738 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275742 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275740 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275765 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275749 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="init-config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275819 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="init-config-reloader" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275833 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-thanos" Jul 29 18:37:22.275841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275841 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-thanos" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275909 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="prom-label-proxy" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275918 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="thanos-sidecar" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275928 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fa0eb56-54b1-48b8-ad6a-696a4763c622" containerName="console" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275937 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="config-reloader" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275946 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="alertmanager" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275952 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275957 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275964 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="dceb0f44-00a6-4508-8415-cbe513796b0c" containerName="console" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275968 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="kube-rbac-proxy-thanos" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275973 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-web" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275979 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275984 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="prometheus" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275992 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" containerName="config-reloader" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.275999 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a405bd-3ebb-412f-a9bc-c6c5b8f44ed0" containerName="kube-rbac-proxy-metric" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276013 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276037 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276254 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276273 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276466 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276486 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276760 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.276919 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.276775 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277017 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277048 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277260 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277275 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277495 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277511 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277787 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.277851 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.277805 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.278125 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278037 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.278125 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278061 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.278301 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278284 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.278345 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278302 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.278526 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278504 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.278613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278526 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.278762 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278744 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.278762 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278761 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.278980 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278960 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.279036 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.278983 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.279213 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279191 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.279213 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279212 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.279427 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279409 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.279470 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279428 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.279654 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279636 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.279708 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279657 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.279889 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279872 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.279934 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279893 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.279934 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.279882 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.280317 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280133 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.280317 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280158 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.280592 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280568 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.280662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280596 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.280847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280824 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.280897 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.280850 2578 scope.go:117] "RemoveContainer" containerID="e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5" Jul 29 18:37:22.281151 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281131 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5"} err="failed to get container status \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": rpc error: code = NotFound desc = could not find container \"e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5\": container with ID starting with e05b9f94dcd2c1bbeb2561df21195cdb3dd1b17f6fac2ae1ac1ed9bb79c9bdd5 not found: ID does not exist" Jul 29 18:37:22.281212 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281152 2578 scope.go:117] "RemoveContainer" containerID="c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654" Jul 29 18:37:22.281365 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281346 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654"} err="failed to get container status \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": rpc error: code = NotFound desc = could not find container \"c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654\": container with ID starting with c4134cddd5cd5e3315bca8e54e1b2dd75f8e32f6d2742118f7ad5b89876a3654 not found: ID does not exist" Jul 29 18:37:22.281410 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281365 2578 scope.go:117] "RemoveContainer" containerID="e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338" Jul 29 18:37:22.281613 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281589 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338"} err="failed to get container status \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": rpc error: code = NotFound desc = could not find container \"e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338\": container with ID starting with e15f3b9fd73e27d3da3216f25e1ddd49e0abcab85567c02f673542de78846338 not found: ID does not exist" Jul 29 18:37:22.281698 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281614 2578 scope.go:117] "RemoveContainer" containerID="cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa" Jul 29 18:37:22.281876 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281859 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa"} err="failed to get container status \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": rpc error: code = NotFound desc = could not find container \"cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa\": container with ID starting with cb2bcba9bb9abd681ce9ab8210744018130b5f03038ac475f9c4d01de5b120aa not found: ID does not exist" Jul 29 18:37:22.281924 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.281876 2578 scope.go:117] "RemoveContainer" containerID="17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb" Jul 29 18:37:22.282137 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282117 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb"} err="failed to get container status \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": rpc error: code = NotFound desc = could not find container \"17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb\": container with ID starting with 17b578f3a7558d576610b72ed0eaa0654ed9c8422aa3fdb3365e963e8acf78bb not found: ID does not exist" Jul 29 18:37:22.282137 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282136 2578 scope.go:117] "RemoveContainer" containerID="8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95" Jul 29 18:37:22.282401 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282382 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95"} err="failed to get container status \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": rpc error: code = NotFound desc = could not find container \"8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95\": container with ID starting with 8e595bd88ef795819fe543d4ef74ae6a765c4664918090bd0ceb6e143d3c4a95 not found: ID does not exist" Jul 29 18:37:22.282468 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282401 2578 scope.go:117] "RemoveContainer" containerID="eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517" Jul 29 18:37:22.282696 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282662 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517"} err="failed to get container status \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": rpc error: code = NotFound desc = could not find container \"eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517\": container with ID starting with eabd72a96e53cca0e21cbd0ad28fe05cee01413a98d339656eca58b53059f517 not found: ID does not exist" Jul 29 18:37:22.282849 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-w9vrs\"" Jul 29 18:37:22.282923 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.282828 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 18:37:22.283035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283011 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 18:37:22.283035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283025 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5agf8kiql0qsk\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283054 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283095 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283029 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283140 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283153 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283154 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 18:37:22.283165 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283171 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 18:37:22.283558 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283329 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 18:37:22.283558 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 18:37:22.283558 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283442 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 18:37:22.283778 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.283742 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 18:37:22.285955 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.285910 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 18:37:22.289392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.289373 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 18:37:22.296627 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.296598 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:22.331168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331130 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331168 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331168 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331311 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331344 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331363 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331392 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331380 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331420 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331512 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331684 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.331751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.331746 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llbc4\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-kube-api-access-llbc4\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.432881 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.432850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.432901 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.432932 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.432970 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.432999 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433028 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433058 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433113 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433176 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433200 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433217 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433242 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433277 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.433298 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433296 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.434299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433334 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.434299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433377 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llbc4\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-kube-api-access-llbc4\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.434299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433419 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.434299 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.433968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.434483 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.434321 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.436579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.436530 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.436767 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.436744 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.437484 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.437464 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.437644 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.437590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.437900 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.437483 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.438002 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.437965 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.438928 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.438468 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.438928 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.438621 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.438928 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.438752 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.439134 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.439009 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.439603 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.439245 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config-out\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.439603 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.439568 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-web-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.439732 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.439680 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.440057 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.440038 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.440508 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.440486 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.441707 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.441689 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-config\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.441790 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.441721 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.445731 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.445708 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llbc4\" (UniqueName: \"kubernetes.io/projected/0ac9c2e4-edf6-4672-9e9d-ccd92677652e-kube-api-access-llbc4\") pod \"prometheus-k8s-0\" (UID: \"0ac9c2e4-edf6-4672-9e9d-ccd92677652e\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.590664 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.590628 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:22.740041 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:22.740015 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:37:22.742697 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:37:22.742664 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ac9c2e4_edf6_4672_9e9d_ccd92677652e.slice/crio-3cc580f6c228a39fc30b6275edb21e618b1ac2f6d36a19937e03b9d0b7c9af1c WatchSource:0}: Error finding container 3cc580f6c228a39fc30b6275edb21e618b1ac2f6d36a19937e03b9d0b7c9af1c: Status 404 returned error can't find the container with id 3cc580f6c228a39fc30b6275edb21e618b1ac2f6d36a19937e03b9d0b7c9af1c Jul 29 18:37:23.217847 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:23.217803 2578 generic.go:356] "Generic (PLEG): container finished" podID="0ac9c2e4-edf6-4672-9e9d-ccd92677652e" containerID="379c7c4278d4825baa46b7a39bc9b319ebb569cc71bc9e58d0fbc3e6a830e2ad" exitCode=0 Jul 29 18:37:23.218063 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:23.217897 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerDied","Data":"379c7c4278d4825baa46b7a39bc9b319ebb569cc71bc9e58d0fbc3e6a830e2ad"} Jul 29 18:37:23.218063 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:23.217979 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"3cc580f6c228a39fc30b6275edb21e618b1ac2f6d36a19937e03b9d0b7c9af1c"} Jul 29 18:37:23.354348 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:23.354248 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9289b45-f6f1-4eff-b322-9336d7b1573f" path="/var/lib/kubelet/pods/a9289b45-f6f1-4eff-b322-9336d7b1573f/volumes" Jul 29 18:37:24.225945 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.225907 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"0530d757978f6428936ae690947dfd74af3f21f493b26c695bc273f7aa79c15c"} Jul 29 18:37:24.226324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.225990 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"ab7c13e05398faaf4ea70b1f0b5bbcf7d824663afdd47f7de1d424acfa425bef"} Jul 29 18:37:24.226324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.226029 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"60e68174e7e4bd9629a55e733ad7dba7d6d4f6b5289c26c00850065fbec611fa"} Jul 29 18:37:24.226324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.226067 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"58eed27a23a4c8d2b144ba951cb76bd35dbd0078a5c0fc43253bbc6a99a5e6fc"} Jul 29 18:37:24.226324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.226106 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"f0a01e2c75ca47a9716216d85a06b7fbd5cbf1bd2f30e58b962769db1e4bd8a2"} Jul 29 18:37:24.226324 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.226141 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0ac9c2e4-edf6-4672-9e9d-ccd92677652e","Type":"ContainerStarted","Data":"b34a5741bb196bace91b6a6f5c1665ca930797c4b19264cd76f0171e3d45f236"} Jul 29 18:37:24.256315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:24.256264 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.236826658 podStartE2EDuration="2.256246451s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.419793ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:37:24.254285217 +0000 UTC m=+161.469450259" watchObservedRunningTime="2026-07-29 18:37:24.256246451 +0000 UTC m=+161.471411492" Jul 29 18:37:27.591683 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:37:27.591625 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:38:22.591623 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:38:22.591575 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:38:22.607588 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:38:22.607538 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:38:23.452067 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:38:23.452036 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:39:43.163083 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:39:43.163050 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:39:43.165856 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:39:43.165834 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:39:43.168330 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:39:43.168313 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 18:41:27.221307 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.221271 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx"] Jul 29 18:41:27.224249 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.224230 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.226799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.226769 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 29 18:41:27.226927 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.226777 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 29 18:41:27.227650 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.227635 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vwthn\"" Jul 29 18:41:27.234211 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.234184 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx"] Jul 29 18:41:27.384807 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.384768 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.384987 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.384817 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dls7\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-kube-api-access-8dls7\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.486122 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.486035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.486122 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.486077 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dls7\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-kube-api-access-8dls7\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.494657 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.494616 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.494780 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.494688 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dls7\" (UniqueName: \"kubernetes.io/projected/1bd68238-b1f2-44f3-8463-efd76c006df7-kube-api-access-8dls7\") pod \"cert-manager-cainjector-559cdd8c87-2ctqx\" (UID: \"1bd68238-b1f2-44f3-8463-efd76c006df7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.534375 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.534321 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" Jul 29 18:41:27.667899 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.667864 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx"] Jul 29 18:41:27.671836 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:41:27.671810 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bd68238_b1f2_44f3_8463_efd76c006df7.slice/crio-09bfbd83367eb061192f7ab8b1e682484d99edd04ed44dc988f4b1a8252b6bb0 WatchSource:0}: Error finding container 09bfbd83367eb061192f7ab8b1e682484d99edd04ed44dc988f4b1a8252b6bb0: Status 404 returned error can't find the container with id 09bfbd83367eb061192f7ab8b1e682484d99edd04ed44dc988f4b1a8252b6bb0 Jul 29 18:41:27.673718 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:27.673705 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:41:28.088834 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:28.088795 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" event={"ID":"1bd68238-b1f2-44f3-8463-efd76c006df7","Type":"ContainerStarted","Data":"09bfbd83367eb061192f7ab8b1e682484d99edd04ed44dc988f4b1a8252b6bb0"} Jul 29 18:41:31.102258 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:31.102220 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" event={"ID":"1bd68238-b1f2-44f3-8463-efd76c006df7","Type":"ContainerStarted","Data":"927ce9f19f1c980fe5984e655f8f7db90076356532f5ca37d99e3f9206307a64"} Jul 29 18:41:31.121779 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:41:31.121729 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-2ctqx" podStartSLOduration=1.068386235 podStartE2EDuration="4.121711967s" totalImagesPullingTime="3.053325732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:41:31.121075973 +0000 UTC m=+408.336241006" watchObservedRunningTime="2026-07-29 18:41:31.121711967 +0000 UTC m=+408.336876995" Jul 29 18:42:02.732959 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.732921 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp"] Jul 29 18:42:02.737478 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.737455 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.741146 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.741124 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 18:42:02.742045 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.742020 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 18:42:02.742175 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.742067 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tllmr\"" Jul 29 18:42:02.742175 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.742106 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:42:02.742311 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.742178 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 18:42:02.742311 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.742208 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 18:42:02.748522 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.748497 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp"] Jul 29 18:42:02.869989 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.869947 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-metrics-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.870170 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.870032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2995352-1f2d-42e8-8db1-2925ac9f658a-manager-config\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.870170 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.870059 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.870170 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.870138 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfp2t\" (UniqueName: \"kubernetes.io/projected/c2995352-1f2d-42e8-8db1-2925ac9f658a-kube-api-access-xfp2t\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.971073 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.971029 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfp2t\" (UniqueName: \"kubernetes.io/projected/c2995352-1f2d-42e8-8db1-2925ac9f658a-kube-api-access-xfp2t\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.971073 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.971074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-metrics-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.971271 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.971135 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2995352-1f2d-42e8-8db1-2925ac9f658a-manager-config\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.971271 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.971166 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.971915 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.971888 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c2995352-1f2d-42e8-8db1-2925ac9f658a-manager-config\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.973778 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.973760 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-metrics-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.973891 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.973818 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c2995352-1f2d-42e8-8db1-2925ac9f658a-cert\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:02.981472 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:02.981442 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfp2t\" (UniqueName: \"kubernetes.io/projected/c2995352-1f2d-42e8-8db1-2925ac9f658a-kube-api-access-xfp2t\") pod \"lws-controller-manager-7698b7b56c-xwcxp\" (UID: \"c2995352-1f2d-42e8-8db1-2925ac9f658a\") " pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:03.048492 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:03.048459 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:03.182210 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:03.182173 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp"] Jul 29 18:42:03.186249 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:42:03.186222 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2995352_1f2d_42e8_8db1_2925ac9f658a.slice/crio-811c4d0ee4d8d828c571bda88d87f49e2f63a63ed919c4ba300a30f5943bdc5c WatchSource:0}: Error finding container 811c4d0ee4d8d828c571bda88d87f49e2f63a63ed919c4ba300a30f5943bdc5c: Status 404 returned error can't find the container with id 811c4d0ee4d8d828c571bda88d87f49e2f63a63ed919c4ba300a30f5943bdc5c Jul 29 18:42:03.219592 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:03.219525 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" event={"ID":"c2995352-1f2d-42e8-8db1-2925ac9f658a","Type":"ContainerStarted","Data":"811c4d0ee4d8d828c571bda88d87f49e2f63a63ed919c4ba300a30f5943bdc5c"} Jul 29 18:42:06.233916 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:06.233887 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" event={"ID":"c2995352-1f2d-42e8-8db1-2925ac9f658a","Type":"ContainerStarted","Data":"ed3c3e1443fca1db12af98b9080c7383a604f21d29330ee7275a49bed57671c1"} Jul 29 18:42:06.234228 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:06.234002 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:06.252813 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:06.252763 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" podStartSLOduration=1.274399045 podStartE2EDuration="4.252747828s" totalImagesPullingTime="2.978348783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:06.251259488 +0000 UTC m=+443.466424518" watchObservedRunningTime="2026-07-29 18:42:06.252747828 +0000 UTC m=+443.467912857" Jul 29 18:42:17.242376 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:17.242346 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7698b7b56c-xwcxp" Jul 29 18:42:25.611921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.611879 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59c47d49bf-r79rd"] Jul 29 18:42:25.615281 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.615254 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.620129 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.620104 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:42:25.620839 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.620815 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:42:25.621493 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.621469 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m998h\"" Jul 29 18:42:25.621640 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.621469 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:42:25.621640 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.621514 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:42:25.621640 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.621588 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:42:25.622227 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.622010 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:42:25.622227 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.622209 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:42:25.625789 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.625756 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:42:25.631571 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.630771 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c47d49bf-r79rd"] Jul 29 18:42:25.666728 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666685 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ctsk\" (UniqueName: \"kubernetes.io/projected/c4aa345e-90d8-4231-a250-8487a47e5e57-kube-api-access-4ctsk\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.666728 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666729 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-console-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.666965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-oauth-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.666965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666809 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.666965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666856 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-service-ca\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.666965 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666872 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-trusted-ca-bundle\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.667150 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.666979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-oauth-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767499 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-oauth-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767499 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767503 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ctsk\" (UniqueName: \"kubernetes.io/projected/c4aa345e-90d8-4231-a250-8487a47e5e57-kube-api-access-4ctsk\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767525 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-console-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767581 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-oauth-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767603 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767632 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-service-ca\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.767771 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.767649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-trusted-ca-bundle\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.768579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.768529 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-oauth-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.768696 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.768583 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-service-ca\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.768696 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.768648 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-trusted-ca-bundle\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.768814 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.768703 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c4aa345e-90d8-4231-a250-8487a47e5e57-console-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.770262 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.770230 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-oauth-config\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.770422 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.770400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c4aa345e-90d8-4231-a250-8487a47e5e57-console-serving-cert\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.778970 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.778946 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ctsk\" (UniqueName: \"kubernetes.io/projected/c4aa345e-90d8-4231-a250-8487a47e5e57-kube-api-access-4ctsk\") pod \"console-59c47d49bf-r79rd\" (UID: \"c4aa345e-90d8-4231-a250-8487a47e5e57\") " pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:25.924902 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:25.924802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:26.063982 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:26.063943 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59c47d49bf-r79rd"] Jul 29 18:42:26.066865 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:42:26.066835 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4aa345e_90d8_4231_a250_8487a47e5e57.slice/crio-0b87ce192d9e0de8d4d9c05c90dc369927913f811dd90bf811e711c6557b5807 WatchSource:0}: Error finding container 0b87ce192d9e0de8d4d9c05c90dc369927913f811dd90bf811e711c6557b5807: Status 404 returned error can't find the container with id 0b87ce192d9e0de8d4d9c05c90dc369927913f811dd90bf811e711c6557b5807 Jul 29 18:42:26.311419 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:26.311377 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c47d49bf-r79rd" event={"ID":"c4aa345e-90d8-4231-a250-8487a47e5e57","Type":"ContainerStarted","Data":"4c7ea04aa27c18542aa76ee9582c87eed2ffd6f83acad3122fb3508bf7f01ff7"} Jul 29 18:42:26.311588 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:26.311447 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59c47d49bf-r79rd" event={"ID":"c4aa345e-90d8-4231-a250-8487a47e5e57","Type":"ContainerStarted","Data":"0b87ce192d9e0de8d4d9c05c90dc369927913f811dd90bf811e711c6557b5807"} Jul 29 18:42:26.331939 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:26.331782 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59c47d49bf-r79rd" podStartSLOduration=1.331759319 podStartE2EDuration="1.331759319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:26.33152687 +0000 UTC m=+463.546691901" watchObservedRunningTime="2026-07-29 18:42:26.331759319 +0000 UTC m=+463.546924349" Jul 29 18:42:35.190777 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.190740 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7"] Jul 29 18:42:35.194906 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.194878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.199893 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.199860 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 18:42:35.200086 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.199934 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 18:42:35.200171 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.200087 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-66ld6\"" Jul 29 18:42:35.214793 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.214755 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7"] Jul 29 18:42:35.247249 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.247211 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeca5f7-cb09-4b16-a149-df4c4699b23a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.247427 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.247332 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lvz\" (UniqueName: \"kubernetes.io/projected/0eeca5f7-cb09-4b16-a149-df4c4699b23a-kube-api-access-m6lvz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.348593 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.348529 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lvz\" (UniqueName: \"kubernetes.io/projected/0eeca5f7-cb09-4b16-a149-df4c4699b23a-kube-api-access-m6lvz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.348798 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.348637 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeca5f7-cb09-4b16-a149-df4c4699b23a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.349019 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.348996 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0eeca5f7-cb09-4b16-a149-df4c4699b23a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.371718 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.371683 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lvz\" (UniqueName: \"kubernetes.io/projected/0eeca5f7-cb09-4b16-a149-df4c4699b23a-kube-api-access-m6lvz\") pod \"kuadrant-operator-controller-manager-66866f5fcc-8cjr7\" (UID: \"0eeca5f7-cb09-4b16-a149-df4c4699b23a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.509763 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.509672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:35.655034 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.654989 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7"] Jul 29 18:42:35.659680 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:42:35.659654 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eeca5f7_cb09_4b16_a149_df4c4699b23a.slice/crio-83395a8a9939ae6eb53784f4ce740c8d4eff290f6f298d329dbcfd2800560d4c WatchSource:0}: Error finding container 83395a8a9939ae6eb53784f4ce740c8d4eff290f6f298d329dbcfd2800560d4c: Status 404 returned error can't find the container with id 83395a8a9939ae6eb53784f4ce740c8d4eff290f6f298d329dbcfd2800560d4c Jul 29 18:42:35.925631 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.925599 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:35.925865 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.925656 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:35.930812 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:35.930773 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:36.350727 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:36.350695 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" event={"ID":"0eeca5f7-cb09-4b16-a149-df4c4699b23a","Type":"ContainerStarted","Data":"83395a8a9939ae6eb53784f4ce740c8d4eff290f6f298d329dbcfd2800560d4c"} Jul 29 18:42:36.354825 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:36.354798 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59c47d49bf-r79rd" Jul 29 18:42:40.370751 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:40.370715 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" event={"ID":"0eeca5f7-cb09-4b16-a149-df4c4699b23a","Type":"ContainerStarted","Data":"749d471c7770d3569ddca02b7b1c5c5e1cf416b20fe917a174eb1047176e6f9a"} Jul 29 18:42:40.371153 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:40.370788 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:42:40.390747 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:40.390685 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" podStartSLOduration=1.254054348 podStartE2EDuration="5.390670038s" totalImagesPullingTime="4.13661569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:40.389822872 +0000 UTC m=+477.604987901" watchObservedRunningTime="2026-07-29 18:42:40.390670038 +0000 UTC m=+477.605835066" Jul 29 18:42:51.376718 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:42:51.376684 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-66866f5fcc-8cjr7" Jul 29 18:43:23.906477 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.906446 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:23.911142 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.911122 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:23.913344 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.913325 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 29 18:43:23.913472 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.913426 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-2dlbv\"" Jul 29 18:43:23.917920 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.917893 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:23.973059 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.973022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:23.973234 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:23.973095 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtl9w\" (UniqueName: \"kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.001182 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.001148 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:24.074093 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.074050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtl9w\" (UniqueName: \"kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.074311 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.074141 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.074877 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.074853 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.082791 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.082761 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtl9w\" (UniqueName: \"kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w\") pod \"limitador-limitador-859c4d96d6-wxlzg\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.223461 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.223364 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:24.366640 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.366500 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:24.369333 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:43:24.369299 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2435a1_ac52_4ca7_93e9_42562032c53c.slice/crio-b315ab07a65e4c1ce2ac238e6c7f1c1acf05186eb9774e7bd7400fcfea256121 WatchSource:0}: Error finding container b315ab07a65e4c1ce2ac238e6c7f1c1acf05186eb9774e7bd7400fcfea256121: Status 404 returned error can't find the container with id b315ab07a65e4c1ce2ac238e6c7f1c1acf05186eb9774e7bd7400fcfea256121 Jul 29 18:43:24.554617 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:24.554580 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" event={"ID":"da2435a1-ac52-4ca7-93e9-42562032c53c","Type":"ContainerStarted","Data":"b315ab07a65e4c1ce2ac238e6c7f1c1acf05186eb9774e7bd7400fcfea256121"} Jul 29 18:43:27.569441 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:27.569400 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" event={"ID":"da2435a1-ac52-4ca7-93e9-42562032c53c","Type":"ContainerStarted","Data":"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0"} Jul 29 18:43:27.569867 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:27.569529 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:27.587858 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:27.587804 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" podStartSLOduration=2.4051858790000002 podStartE2EDuration="4.587787016s" totalImagesPullingTime="2.182601137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:27.587156642 +0000 UTC m=+524.802321685" watchObservedRunningTime="2026-07-29 18:43:27.587787016 +0000 UTC m=+524.802952073" Jul 29 18:43:37.893729 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:37.893689 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:37.894217 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:37.893901 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" podUID="da2435a1-ac52-4ca7-93e9-42562032c53c" containerName="limitador" containerID="cri-o://0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0" gracePeriod=30 Jul 29 18:43:37.894712 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:37.894561 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:38.446925 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.446899 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:38.508114 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.508021 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file\") pod \"da2435a1-ac52-4ca7-93e9-42562032c53c\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " Jul 29 18:43:38.508280 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.508142 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtl9w\" (UniqueName: \"kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w\") pod \"da2435a1-ac52-4ca7-93e9-42562032c53c\" (UID: \"da2435a1-ac52-4ca7-93e9-42562032c53c\") " Jul 29 18:43:38.508386 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.508361 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file" (OuterVolumeSpecName: "config-file") pod "da2435a1-ac52-4ca7-93e9-42562032c53c" (UID: "da2435a1-ac52-4ca7-93e9-42562032c53c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:43:38.510557 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.510514 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w" (OuterVolumeSpecName: "kube-api-access-mtl9w") pod "da2435a1-ac52-4ca7-93e9-42562032c53c" (UID: "da2435a1-ac52-4ca7-93e9-42562032c53c"). InnerVolumeSpecName "kube-api-access-mtl9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:38.609045 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.609002 2578 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/da2435a1-ac52-4ca7-93e9-42562032c53c-config-file\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:43:38.609045 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.609037 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtl9w\" (UniqueName: \"kubernetes.io/projected/da2435a1-ac52-4ca7-93e9-42562032c53c-kube-api-access-mtl9w\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:43:38.619674 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.619645 2578 generic.go:356] "Generic (PLEG): container finished" podID="da2435a1-ac52-4ca7-93e9-42562032c53c" containerID="0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0" exitCode=0 Jul 29 18:43:38.619818 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.619714 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" Jul 29 18:43:38.619818 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.619737 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" event={"ID":"da2435a1-ac52-4ca7-93e9-42562032c53c","Type":"ContainerDied","Data":"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0"} Jul 29 18:43:38.619818 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.619812 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-859c4d96d6-wxlzg" event={"ID":"da2435a1-ac52-4ca7-93e9-42562032c53c","Type":"ContainerDied","Data":"b315ab07a65e4c1ce2ac238e6c7f1c1acf05186eb9774e7bd7400fcfea256121"} Jul 29 18:43:38.619918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.619838 2578 scope.go:117] "RemoveContainer" containerID="0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0" Jul 29 18:43:38.628626 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.628606 2578 scope.go:117] "RemoveContainer" containerID="0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0" Jul 29 18:43:38.628896 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:43:38.628877 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0\": container with ID starting with 0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0 not found: ID does not exist" containerID="0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0" Jul 29 18:43:38.628942 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.628906 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0"} err="failed to get container status \"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0\": rpc error: code = NotFound desc = could not find container \"0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0\": container with ID starting with 0b0440cbd8b8e591dbb862205be0cce85d2102cc6d77c2c71f72b5dd7ff77da0 not found: ID does not exist" Jul 29 18:43:38.640741 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.640711 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:38.646792 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:38.646766 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-859c4d96d6-wxlzg"] Jul 29 18:43:39.354056 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:39.354019 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da2435a1-ac52-4ca7-93e9-42562032c53c" path="/var/lib/kubelet/pods/da2435a1-ac52-4ca7-93e9-42562032c53c/volumes" Jul 29 18:43:58.601361 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.601305 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc"] Jul 29 18:43:58.602032 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.602010 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da2435a1-ac52-4ca7-93e9-42562032c53c" containerName="limitador" Jul 29 18:43:58.602107 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.602035 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da2435a1-ac52-4ca7-93e9-42562032c53c" containerName="limitador" Jul 29 18:43:58.602180 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.602158 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da2435a1-ac52-4ca7-93e9-42562032c53c" containerName="limitador" Jul 29 18:43:58.607857 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.607834 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.613308 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.613282 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 18:43:58.613479 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.613284 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 18:43:58.613479 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.613286 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 18:43:58.613893 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.613877 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-94q2z\"" Jul 29 18:43:58.614515 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.614432 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 18:43:58.614713 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.614695 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 18:43:58.614802 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.614696 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 18:43:58.656376 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.656346 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc"] Jul 29 18:43:58.680337 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680303 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680337 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680375 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680417 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c5b6c9-adbc-408b-9956-5122d977d12f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c7df\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-kube-api-access-5c7df\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.680579 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.680477 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.780827 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780889 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780909 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c5b6c9-adbc-408b-9956-5122d977d12f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780924 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c7df\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-kube-api-access-5c7df\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781025 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.780943 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.781943 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.781917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.783576 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.783522 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.783681 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.783534 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.783681 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.783590 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.783681 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.783592 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/12c5b6c9-adbc-408b-9956-5122d977d12f-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.792897 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.792868 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.793315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.793291 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c7df\" (UniqueName: \"kubernetes.io/projected/12c5b6c9-adbc-408b-9956-5122d977d12f-kube-api-access-5c7df\") pod \"istiod-openshift-gateway-5677ccb49b-9fbfc\" (UID: \"12c5b6c9-adbc-408b-9956-5122d977d12f\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:58.917305 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:58.917212 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:43:59.092433 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:59.092405 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc"] Jul 29 18:43:59.095261 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:43:59.095235 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c5b6c9_adbc_408b_9956_5122d977d12f.slice/crio-a4884b6d763de7d5d09eacccf614b01a86d85cbff5db01393d0ed951fbc3ed6e WatchSource:0}: Error finding container a4884b6d763de7d5d09eacccf614b01a86d85cbff5db01393d0ed951fbc3ed6e: Status 404 returned error can't find the container with id a4884b6d763de7d5d09eacccf614b01a86d85cbff5db01393d0ed951fbc3ed6e Jul 29 18:43:59.707707 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:43:59.707667 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" event={"ID":"12c5b6c9-adbc-408b-9956-5122d977d12f","Type":"ContainerStarted","Data":"a4884b6d763de7d5d09eacccf614b01a86d85cbff5db01393d0ed951fbc3ed6e"} Jul 29 18:44:05.731971 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:05.731931 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 18:44:05.732274 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:05.731997 2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 29 18:44:06.736830 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:06.736792 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" event={"ID":"12c5b6c9-adbc-408b-9956-5122d977d12f","Type":"ContainerStarted","Data":"17e97cd6c0a941815c013f933f0590c745b3839cf295006a66d67ae284c665f8"} Jul 29 18:44:06.737315 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:06.736849 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:44:06.738675 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:06.738652 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" Jul 29 18:44:06.759438 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:06.759387 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-9fbfc" podStartSLOduration=2.124916382 podStartE2EDuration="8.759370072s" totalImagesPullingTime="6.63445369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:06.757419332 +0000 UTC m=+563.972584363" watchObservedRunningTime="2026-07-29 18:44:06.759370072 +0000 UTC m=+563.974535101" Jul 29 18:44:43.188665 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:43.188633 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:44:43.189162 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:44:43.189070 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:46:27.469779 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.469747 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:27.472087 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.472068 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.475650 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.475627 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 29 18:46:27.475779 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.475661 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"gw-sec0c69dceeb48768325d1a53a749e65786-kserve-self-signed-certs\"" Jul 29 18:46:27.475779 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.475675 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\"" Jul 29 18:46:27.475779 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.475675 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 29 18:46:27.484999 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.484977 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:27.571816 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.571781 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.571816 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.571821 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.572035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.571892 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.572035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.571926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc49p\" (UniqueName: \"kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.572035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.571979 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.572035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.572013 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.572164 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.572040 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673266 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673233 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673266 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673505 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673505 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673322 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc49p\" (UniqueName: \"kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673505 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673505 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673505 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673403 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673821 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673791 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673872 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673804 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673938 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673917 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.673995 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.673968 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.676266 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.676233 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.676623 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.676603 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.682500 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.682475 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc49p\" (UniqueName: \"kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p\") pod \"gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.782208 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.782172 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:27.919099 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.919062 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:27.922256 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:46:27.922233 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7b92f7_232a_46fe_b18f_ff2f6ec87c6e.slice/crio-19e5f64ab8b17e7d63e954a0697266f982b8d35a2400112d29dd3efb19c4e50b WatchSource:0}: Error finding container 19e5f64ab8b17e7d63e954a0697266f982b8d35a2400112d29dd3efb19c4e50b: Status 404 returned error can't find the container with id 19e5f64ab8b17e7d63e954a0697266f982b8d35a2400112d29dd3efb19c4e50b Jul 29 18:46:27.924581 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:27.924560 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:46:28.316649 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:28.316610 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerStarted","Data":"19e5f64ab8b17e7d63e954a0697266f982b8d35a2400112d29dd3efb19c4e50b"} Jul 29 18:46:32.339861 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:32.339815 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerStarted","Data":"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77"} Jul 29 18:46:38.373625 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:38.373588 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerID="1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77" exitCode=0 Jul 29 18:46:38.374061 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:38.373665 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerDied","Data":"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77"} Jul 29 18:46:38.646149 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:38.645973 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:40.384295 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.384256 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerStarted","Data":"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f"} Jul 29 18:46:40.384794 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.384362 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="main" containerID="cri-o://644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f" gracePeriod=30 Jul 29 18:46:40.404708 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.404662 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" podStartSLOduration=2.4810886979999998 podStartE2EDuration="13.404645179s" totalImagesPullingTime="4.31840083s" totalInitContainerRuntime="6.605155651s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:46:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:46:40.402755619 +0000 UTC m=+717.617920651" watchObservedRunningTime="2026-07-29 18:46:40.404645179 +0000 UTC m=+717.619810207" Jul 29 18:46:40.640726 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.640669 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:40.697585 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697532 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc49p\" (UniqueName: \"kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697753 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697606 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697753 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697635 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697753 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697662 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697758 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697783 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697826 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache\") pod \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\" (UID: \"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e\") " Jul 29 18:46:40.697918 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697909 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home" (OuterVolumeSpecName: "home") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:46:40.698130 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.697962 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:46:40.698130 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.698091 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:46:40.698130 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.698118 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-home\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.698438 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.698133 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tmp-dir\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.700055 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.700023 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm" (OuterVolumeSpecName: "dshm") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:46:40.700055 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.700035 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p" (OuterVolumeSpecName: "kube-api-access-rc49p") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "kube-api-access-rc49p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:46:40.700240 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.700103 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:46:40.762402 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.762355 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" (UID: "6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:46:40.799462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.799427 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc49p\" (UniqueName: \"kubernetes.io/projected/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kube-api-access-rc49p\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.799462 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.799458 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-dshm\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.799658 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.799470 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-tls-certs\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.799658 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.799485 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-kserve-provision-location\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:40.799658 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:40.799494 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e-model-cache\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:46:41.389729 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.389691 2578 generic.go:356] "Generic (PLEG): container finished" podID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerID="644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f" exitCode=0 Jul 29 18:46:41.390133 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.389754 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" Jul 29 18:46:41.390133 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.389763 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerDied","Data":"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f"} Jul 29 18:46:41.390133 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.389804 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h" event={"ID":"6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e","Type":"ContainerDied","Data":"19e5f64ab8b17e7d63e954a0697266f982b8d35a2400112d29dd3efb19c4e50b"} Jul 29 18:46:41.390133 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.389826 2578 scope.go:117] "RemoveContainer" containerID="644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f" Jul 29 18:46:41.398370 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.398346 2578 scope.go:117] "RemoveContainer" containerID="1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77" Jul 29 18:46:41.409263 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.409240 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:41.413597 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.413574 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/gw-section-name-router-with-gat-2f0a622e-kserve-84d895999cfw22h"] Jul 29 18:46:41.468167 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.468144 2578 scope.go:117] "RemoveContainer" containerID="644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f" Jul 29 18:46:41.468539 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:46:41.468520 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f\": container with ID starting with 644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f not found: ID does not exist" containerID="644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f" Jul 29 18:46:41.468609 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.468563 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f"} err="failed to get container status \"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f\": rpc error: code = NotFound desc = could not find container \"644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f\": container with ID starting with 644b3cac9fd1da5488aeda6ec218af1789d94a83e36c5dfdf0715a2ce2ad6a6f not found: ID does not exist" Jul 29 18:46:41.468609 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.468583 2578 scope.go:117] "RemoveContainer" containerID="1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77" Jul 29 18:46:41.469039 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:46:41.469022 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77\": container with ID starting with 1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77 not found: ID does not exist" containerID="1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77" Jul 29 18:46:41.469080 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:41.469051 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77"} err="failed to get container status \"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77\": rpc error: code = NotFound desc = could not find container \"1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77\": container with ID starting with 1fd5d66a66230c420b055dbbfecb603bc415801a20e26a9d5aab8757e1b78c77 not found: ID does not exist" Jul 29 18:46:43.354499 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:46:43.354462 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" path="/var/lib/kubelet/pods/6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e/volumes" Jul 29 18:47:10.369652 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.369604 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:47:10.370126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.369938 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="main" Jul 29 18:47:10.370126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.369953 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="main" Jul 29 18:47:10.370126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.369966 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="storage-initializer" Jul 29 18:47:10.370126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.369971 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="storage-initializer" Jul 29 18:47:10.370126 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.370031 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f7b92f7-232a-46fe-b18f-ff2f6ec87c6e" containerName="main" Jul 29 18:47:10.372109 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.372082 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.374664 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.374642 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\"" Jul 29 18:47:10.374789 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.374682 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 29 18:47:10.375662 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.375646 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 29 18:47:10.375727 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.375709 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvad71fa5348b85aebd404221bba611457-kserve-self-signed-certs\"" Jul 29 18:47:10.382097 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.382067 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:47:10.473352 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473319 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473352 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m285s\" (UniqueName: \"kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473425 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473471 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.473671 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.473641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574590 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574648 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574664 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574720 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m285s\" (UniqueName: \"kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.574799 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.574765 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.575156 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.575122 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.575220 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.575129 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.575220 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.575181 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.575220 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.575206 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.577034 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.577011 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.577256 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.577241 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.582423 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.582400 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m285s\" (UniqueName: \"kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s\") pod \"llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.683256 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.683166 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:10.826190 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:10.826163 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:47:10.829137 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:47:10.829102 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacba2afb_349d_42ba_ae84_f50b0414920b.slice/crio-a089340335fc4c5d844dafa18113d48859dad8fc1e9846319deaa564905ff703 WatchSource:0}: Error finding container a089340335fc4c5d844dafa18113d48859dad8fc1e9846319deaa564905ff703: Status 404 returned error can't find the container with id a089340335fc4c5d844dafa18113d48859dad8fc1e9846319deaa564905ff703 Jul 29 18:47:11.515054 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:11.515019 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerStarted","Data":"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706"} Jul 29 18:47:11.515436 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:11.515118 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerStarted","Data":"a089340335fc4c5d844dafa18113d48859dad8fc1e9846319deaa564905ff703"} Jul 29 18:47:17.542998 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:17.542963 2578 generic.go:356] "Generic (PLEG): container finished" podID="acba2afb-349d-42ba-ae84-f50b0414920b" containerID="a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706" exitCode=0 Jul 29 18:47:17.543386 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:17.543034 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerDied","Data":"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706"} Jul 29 18:47:18.549357 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:18.549322 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerStarted","Data":"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced"} Jul 29 18:47:18.569360 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:18.569304 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" podStartSLOduration=2.790795859 podStartE2EDuration="8.56928749s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.778491631s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:47:18.56775885 +0000 UTC m=+755.782923893" watchObservedRunningTime="2026-07-29 18:47:18.56928749 +0000 UTC m=+755.784452519" Jul 29 18:47:20.684023 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:20.683985 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:20.684439 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:20.684152 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:20.697169 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:20.696722 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:47:21.573468 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:47:21.573440 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:49:08.456039 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.456004 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:08.458448 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.458424 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.460850 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.460825 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\"" Jul 29 18:49:08.470738 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.470709 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:08.636152 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636108 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636331 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636162 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636331 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636191 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636331 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636301 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ww22\" (UniqueName: \"kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636433 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636334 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636433 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.636433 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.636399 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.737877 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.737777 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738054 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.737938 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738054 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.737992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738054 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738219 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ww22\" (UniqueName: \"kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738219 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738138 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738219 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738399 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738373 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738460 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738407 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738656 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738594 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.738656 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.738630 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.740278 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.740263 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.740429 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.740414 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.746394 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.746365 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ww22\" (UniqueName: \"kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22\") pod \"precise-prefix-cache-test-kserve-79764f4d79-fp86t\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.769143 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.769102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:08.911050 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:08.911025 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:08.913514 ip-10-0-129-174 kubenswrapper[2578]: W0729 18:49:08.913483 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2346acb_ec6a_4f82_bef7_0d3550ca9966.slice/crio-b496fa3aa03590e7a35702031eed2fe341692caae13c7d244a1810aa4ae0a9de WatchSource:0}: Error finding container b496fa3aa03590e7a35702031eed2fe341692caae13c7d244a1810aa4ae0a9de: Status 404 returned error can't find the container with id b496fa3aa03590e7a35702031eed2fe341692caae13c7d244a1810aa4ae0a9de Jul 29 18:49:09.019536 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:09.019444 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerStarted","Data":"8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f"} Jul 29 18:49:09.019536 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:09.019501 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerStarted","Data":"b496fa3aa03590e7a35702031eed2fe341692caae13c7d244a1810aa4ae0a9de"} Jul 29 18:49:15.050313 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:15.050277 2578 generic.go:356] "Generic (PLEG): container finished" podID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerID="8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f" exitCode=0 Jul 29 18:49:15.050735 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:15.050336 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerDied","Data":"8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f"} Jul 29 18:49:16.056561 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:16.056508 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerStarted","Data":"61c46d9e86fb969751778b6cc7c9b9f51e1b97bb3491ee114583c7054ff48519"} Jul 29 18:49:16.076563 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:16.076489 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" podStartSLOduration=2.676578379 podStartE2EDuration="8.07647091s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.399892531s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:49:16.075240052 +0000 UTC m=+873.290405082" watchObservedRunningTime="2026-07-29 18:49:16.07647091 +0000 UTC m=+873.291635938" Jul 29 18:49:18.769271 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:18.769231 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:18.769696 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:18.769306 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:18.782666 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:18.782637 2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:19.081405 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:19.081377 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:41.974180 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:41.974142 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:41.974828 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:41.974373 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="main" containerID="cri-o://61c46d9e86fb969751778b6cc7c9b9f51e1b97bb3491ee114583c7054ff48519" gracePeriod=30 Jul 29 18:49:42.178035 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.178006 2578 generic.go:356] "Generic (PLEG): container finished" podID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerID="61c46d9e86fb969751778b6cc7c9b9f51e1b97bb3491ee114583c7054ff48519" exitCode=0 Jul 29 18:49:42.178186 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.178074 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerDied","Data":"61c46d9e86fb969751778b6cc7c9b9f51e1b97bb3491ee114583c7054ff48519"} Jul 29 18:49:42.235534 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.235476 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:42.339796 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339762 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.339969 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339819 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.339969 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339869 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.339969 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339905 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.339969 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339947 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.340151 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.339972 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ww22\" (UniqueName: \"kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.340151 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340012 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location\") pod \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\" (UID: \"e2346acb-ec6a-4f82-bef7-0d3550ca9966\") " Jul 29 18:49:42.340151 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340065 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache" (OuterVolumeSpecName: "model-cache") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:49:42.340373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340143 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:49:42.340373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340219 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home" (OuterVolumeSpecName: "home") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:49:42.340373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340307 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-home\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.340373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340322 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-model-cache\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.340373 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.340354 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tmp-dir\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.342270 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.342238 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm" (OuterVolumeSpecName: "dshm") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:49:42.342270 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.342264 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:49:42.342422 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.342389 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22" (OuterVolumeSpecName: "kube-api-access-5ww22") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "kube-api-access-5ww22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:49:42.395606 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.395561 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e2346acb-ec6a-4f82-bef7-0d3550ca9966" (UID: "e2346acb-ec6a-4f82-bef7-0d3550ca9966"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:49:42.441733 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.441706 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2346acb-ec6a-4f82-bef7-0d3550ca9966-tls-certs\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.441733 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.441732 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-dshm\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.441733 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.441744 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ww22\" (UniqueName: \"kubernetes.io/projected/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kube-api-access-5ww22\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:42.441978 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:42.441760 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e2346acb-ec6a-4f82-bef7-0d3550ca9966-kserve-provision-location\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:49:43.184581 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.184530 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" Jul 29 18:49:43.184581 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.184564 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t" event={"ID":"e2346acb-ec6a-4f82-bef7-0d3550ca9966","Type":"ContainerDied","Data":"b496fa3aa03590e7a35702031eed2fe341692caae13c7d244a1810aa4ae0a9de"} Jul 29 18:49:43.185138 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.184643 2578 scope.go:117] "RemoveContainer" containerID="61c46d9e86fb969751778b6cc7c9b9f51e1b97bb3491ee114583c7054ff48519" Jul 29 18:49:43.193475 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.193451 2578 scope.go:117] "RemoveContainer" containerID="8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f" Jul 29 18:49:43.208243 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:49:43.208212 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f\": container with ID starting with 8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f not found: ID does not exist" containerID="8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f" Jul 29 18:49:43.208428 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:49:43.208258 2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f\": container with ID starting with 8a55204408a0faf2900e3d14695ecbe4ac8d34f0136b7f52fd29f8da5fe4206f not found: ID does not exist" Jul 29 18:49:43.208523 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.208498 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:43.214358 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.214326 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-79764f4d79-fp86t"] Jul 29 18:49:43.219841 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.219816 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:49:43.354499 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:49:43.354463 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" path="/var/lib/kubelet/pods/e2346acb-ec6a-4f82-bef7-0d3550ca9966/volumes" Jul 29 18:51:27.673798 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:27.673756 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:51:27.674314 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:27.674008 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="main" containerID="cri-o://98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced" gracePeriod=30 Jul 29 18:51:27.925159 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:27.925096 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:51:28.086290 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086252 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086491 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086308 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086491 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086337 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086491 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086386 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m285s\" (UniqueName: \"kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086491 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086459 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086493 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086520 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm\") pod \"acba2afb-349d-42ba-ae84-f50b0414920b\" (UID: \"acba2afb-349d-42ba-ae84-f50b0414920b\") " Jul 29 18:51:28.086742 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086696 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home" (OuterVolumeSpecName: "home") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:51:28.086892 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086819 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:51:28.086892 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086833 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache" (OuterVolumeSpecName: "model-cache") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:51:28.086892 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.086879 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-home\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.088780 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.088752 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:51:28.088780 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.088765 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm" (OuterVolumeSpecName: "dshm") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:51:28.088921 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.088825 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s" (OuterVolumeSpecName: "kube-api-access-m285s") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "kube-api-access-m285s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:51:28.144639 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.144596 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "acba2afb-349d-42ba-ae84-f50b0414920b" (UID: "acba2afb-349d-42ba-ae84-f50b0414920b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187757 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m285s\" (UniqueName: \"kubernetes.io/projected/acba2afb-349d-42ba-ae84-f50b0414920b-kube-api-access-m285s\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187793 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-tmp-dir\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187810 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-model-cache\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187822 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-dshm\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187835 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/acba2afb-349d-42ba-ae84-f50b0414920b-tls-certs\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.187843 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.187843 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/acba2afb-349d-42ba-ae84-f50b0414920b-kserve-provision-location\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 18:51:28.634200 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.634164 2578 generic.go:356] "Generic (PLEG): container finished" podID="acba2afb-349d-42ba-ae84-f50b0414920b" containerID="98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced" exitCode=0 Jul 29 18:51:28.634383 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.634275 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" Jul 29 18:51:28.634383 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.634293 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerDied","Data":"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced"} Jul 29 18:51:28.634383 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.634345 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8" event={"ID":"acba2afb-349d-42ba-ae84-f50b0414920b","Type":"ContainerDied","Data":"a089340335fc4c5d844dafa18113d48859dad8fc1e9846319deaa564905ff703"} Jul 29 18:51:28.634383 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.634373 2578 scope.go:117] "RemoveContainer" containerID="98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced" Jul 29 18:51:28.643772 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.643754 2578 scope.go:117] "RemoveContainer" containerID="a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706" Jul 29 18:51:28.653844 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.653824 2578 scope.go:117] "RemoveContainer" containerID="98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced" Jul 29 18:51:28.654159 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:51:28.654140 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced\": container with ID starting with 98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced not found: ID does not exist" containerID="98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced" Jul 29 18:51:28.654208 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.654168 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced"} err="failed to get container status \"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced\": rpc error: code = NotFound desc = could not find container \"98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced\": container with ID starting with 98077cf293f4c119ad35d535838b3111e03d9081f888e1121ad71f12542e5ced not found: ID does not exist" Jul 29 18:51:28.654208 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.654187 2578 scope.go:117] "RemoveContainer" containerID="a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706" Jul 29 18:51:28.654447 ip-10-0-129-174 kubenswrapper[2578]: E0729 18:51:28.654430 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706\": container with ID starting with a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706 not found: ID does not exist" containerID="a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706" Jul 29 18:51:28.654493 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.654452 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706"} err="failed to get container status \"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706\": rpc error: code = NotFound desc = could not find container \"a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706\": container with ID starting with a5f8f4aee6531a77b106a2cba9a2a765d0a809e51f307e75ac290c5e6a867706 not found: ID does not exist" Jul 29 18:51:28.656769 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.656746 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:51:28.659686 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:28.659659 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-6fa1027a-kserve-f78d7988b-lzvn8"] Jul 29 18:51:29.354100 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:51:29.354062 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" path="/var/lib/kubelet/pods/acba2afb-349d-42ba-ae84-f50b0414920b/volumes" Jul 29 18:54:43.227483 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:54:43.227446 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:54:43.233145 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:54:43.233118 2578 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 18:54:43.249011 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:54:43.248982 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:59:43.251327 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:59:43.251299 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 18:59:43.271948 ip-10-0-129-174 kubenswrapper[2578]: I0729 18:59:43.271923 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:04:43.276283 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:04:43.276254 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:04:43.293842 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:04:43.293817 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:09:43.302443 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:09:43.302413 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:09:43.321816 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:09:43.321787 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:14:43.327869 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:14:43.327761 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:14:43.343906 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:14:43.343873 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:15:07.511204 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511121 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511454 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="storage-initializer" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511466 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="storage-initializer" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511481 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="storage-initializer" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511486 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="storage-initializer" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511493 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="main" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511499 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="main" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511515 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="main" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511520 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="main" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511587 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2346acb-ec6a-4f82-bef7-0d3550ca9966" containerName="main" Jul 29 19:15:07.511662 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.511598 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="acba2afb-349d-42ba-ae84-f50b0414920b" containerName="main" Jul 29 19:15:07.514412 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.514393 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.518785 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.518758 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\"" Jul 29 19:15:07.521380 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.521040 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\"" Jul 29 19:15:07.521380 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.521055 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv4e643bc258191ffc517a31cd1d0ddd27-kserve-self-signed-certs\"" Jul 29 19:15:07.521380 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.521240 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-f312f5-cb7fb8cf-dockercfg-dm5z2\"" Jul 29 19:15:07.521380 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.521317 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2cmt9\"" Jul 29 19:15:07.531812 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.531774 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:15:07.635201 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635166 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635201 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635203 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635453 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635232 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635453 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635249 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635453 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635320 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635453 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.635453 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.635395 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9ljx\" (UniqueName: \"kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736320 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736320 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736323 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736593 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736349 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736593 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736593 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736411 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9ljx\" (UniqueName: \"kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736593 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736443 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736593 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736885 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736772 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736885 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736794 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.736885 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.736835 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.737027 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.737013 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.738882 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.738860 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.739078 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.739062 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.745895 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.745874 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9ljx\" (UniqueName: \"kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx\") pod \"llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.824570 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.824509 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:15:07.980686 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:07.980515 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:15:07.983010 ip-10-0-129-174 kubenswrapper[2578]: W0729 19:15:07.982983 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b353b3a_f999_4f7c_a448_0f8aa9367213.slice/crio-e883ce20dbe8463471580dff6dc1927a2d230755d377117b71e9f61380094d98 WatchSource:0}: Error finding container e883ce20dbe8463471580dff6dc1927a2d230755d377117b71e9f61380094d98: Status 404 returned error can't find the container with id e883ce20dbe8463471580dff6dc1927a2d230755d377117b71e9f61380094d98 Jul 29 19:15:08.479795 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:08.479754 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerStarted","Data":"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128"} Jul 29 19:15:08.480014 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:08.479826 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerStarted","Data":"e883ce20dbe8463471580dff6dc1927a2d230755d377117b71e9f61380094d98"} Jul 29 19:15:13.505786 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:13.505687 2578 generic.go:356] "Generic (PLEG): container finished" podID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerID="84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128" exitCode=0 Jul 29 19:15:13.505786 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:13.505770 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerDied","Data":"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128"} Jul 29 19:15:13.507222 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:13.507204 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:15:52.712457 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:52.712415 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerStarted","Data":"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673"} Jul 29 19:15:52.734478 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:15:52.734414 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podStartSLOduration=2.11219663 podStartE2EDuration="45.734392541s" totalImagesPullingTime="38.579426713s" totalInitContainerRuntime="5.042769198s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:52.73172931 +0000 UTC m=+2469.946894340" watchObservedRunningTime="2026-07-29 19:15:52.734392541 +0000 UTC m=+2469.949557573" Jul 29 19:17:37.120976 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:37.120935 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:17:37.121833 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:37.121243 2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="main" containerID="cri-o://0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673" gracePeriod=30 Jul 29 19:17:37.876071 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:37.876045 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:17:38.010620 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010503 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010620 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010594 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9ljx\" (UniqueName: \"kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010849 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010631 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010849 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010691 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010849 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010728 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010849 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010746 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.010849 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.010765 2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir\") pod \"6b353b3a-f999-4f7c-a448-0f8aa9367213\" (UID: \"6b353b3a-f999-4f7c-a448-0f8aa9367213\") " Jul 29 19:17:38.011377 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.011300 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache" (OuterVolumeSpecName: "model-cache") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:17:38.011493 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.011426 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home" (OuterVolumeSpecName: "home") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:17:38.013361 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.013331 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm" (OuterVolumeSpecName: "dshm") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:17:38.013577 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.013530 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:17:38.013667 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.013578 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx" (OuterVolumeSpecName: "kube-api-access-z9ljx") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "kube-api-access-z9ljx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:17:38.022805 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.022766 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:17:38.073046 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.072999 2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b353b3a-f999-4f7c-a448-0f8aa9367213" (UID: "6b353b3a-f999-4f7c-a448-0f8aa9367213"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:17:38.112433 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112399 2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-dshm\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112433 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112428 2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-model-cache\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112433 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112437 2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-tmp-dir\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112688 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112447 2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-home\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112688 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112455 2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9ljx\" (UniqueName: \"kubernetes.io/projected/6b353b3a-f999-4f7c-a448-0f8aa9367213-kube-api-access-z9ljx\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112688 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112463 2578 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b353b3a-f999-4f7c-a448-0f8aa9367213-kserve-provision-location\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.112688 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.112471 2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b353b3a-f999-4f7c-a448-0f8aa9367213-tls-certs\") on node \"ip-10-0-129-174.ec2.internal\" DevicePath \"\"" Jul 29 19:17:38.182745 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.182708 2578 generic.go:356] "Generic (PLEG): container finished" podID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerID="0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673" exitCode=0 Jul 29 19:17:38.183137 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.182764 2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" Jul 29 19:17:38.183137 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.182784 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerDied","Data":"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673"} Jul 29 19:17:38.183137 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.182828 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1" event={"ID":"6b353b3a-f999-4f7c-a448-0f8aa9367213","Type":"ContainerDied","Data":"e883ce20dbe8463471580dff6dc1927a2d230755d377117b71e9f61380094d98"} Jul 29 19:17:38.183137 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.182850 2578 scope.go:117] "RemoveContainer" containerID="0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673" Jul 29 19:17:38.192408 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.192385 2578 scope.go:117] "RemoveContainer" containerID="84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128" Jul 29 19:17:38.202579 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.202558 2578 scope.go:117] "RemoveContainer" containerID="0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673" Jul 29 19:17:38.202921 ip-10-0-129-174 kubenswrapper[2578]: E0729 19:17:38.202903 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673\": container with ID starting with 0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673 not found: ID does not exist" containerID="0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673" Jul 29 19:17:38.202973 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.202930 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673"} err="failed to get container status \"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673\": rpc error: code = NotFound desc = could not find container \"0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673\": container with ID starting with 0ae5c974a44de9b4f28fde7870a11a8f93218a80ad34705b3522218926a76673 not found: ID does not exist" Jul 29 19:17:38.202973 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.202949 2578 scope.go:117] "RemoveContainer" containerID="84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128" Jul 29 19:17:38.203181 ip-10-0-129-174 kubenswrapper[2578]: E0729 19:17:38.203161 2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128\": container with ID starting with 84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128 not found: ID does not exist" containerID="84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128" Jul 29 19:17:38.203243 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.203192 2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128"} err="failed to get container status \"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128\": rpc error: code = NotFound desc = could not find container \"84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128\": container with ID starting with 84254445d36b04d07aa02f7fef9f5dcab44458f780d102249ede9f1ea9d10128 not found: ID does not exist" Jul 29 19:17:38.206974 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.206949 2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:17:38.211511 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:38.211479 2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-f312f5ec-kserve-mn-0-1"] Jul 29 19:17:39.353782 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:17:39.353746 2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" path="/var/lib/kubelet/pods/6b353b3a-f999-4f7c-a448-0f8aa9367213/volumes" Jul 29 19:18:08.496405 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:08.496320 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9fbfc_12c5b6c9-adbc-408b-9956-5122d977d12f/discovery/0.log" Jul 29 19:18:09.412298 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:09.412248 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9fbfc_12c5b6c9-adbc-408b-9956-5122d977d12f/discovery/0.log" Jul 29 19:18:10.381095 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:10.381062 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-8cjr7_0eeca5f7-cb09-4b16-a149-df4c4699b23a/manager/0.log" Jul 29 19:18:15.965212 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:15.965175 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2vwxh_a610eee9-34c8-4d8f-82df-a0405f0c7ff3/global-pull-secret-syncer/0.log" Jul 29 19:18:16.223219 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:16.223137 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dbhmw_33461b5b-deeb-4b99-af20-390ad7ccdac6/konnectivity-agent/0.log" Jul 29 19:18:16.276885 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:16.276855 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-174.ec2.internal_eb52c0df565696781869de4afb3c91b0/haproxy/0.log" Jul 29 19:18:20.491332 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:20.491301 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-66866f5fcc-8cjr7_0eeca5f7-cb09-4b16-a149-df4c4699b23a/manager/0.log" Jul 29 19:18:22.400359 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.400323 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lbnbm_9357e329-51f2-4227-ac96-e73e3e81a0d6/kube-state-metrics/0.log" Jul 29 19:18:22.436793 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.436756 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lbnbm_9357e329-51f2-4227-ac96-e73e3e81a0d6/kube-rbac-proxy-main/0.log" Jul 29 19:18:22.462863 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.462833 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lbnbm_9357e329-51f2-4227-ac96-e73e3e81a0d6/kube-rbac-proxy-self/0.log" Jul 29 19:18:22.572155 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.572128 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24z2z_38f8b54c-882f-4036-be37-7f42cdbc0f5d/node-exporter/0.log" Jul 29 19:18:22.602271 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.602247 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24z2z_38f8b54c-882f-4036-be37-7f42cdbc0f5d/kube-rbac-proxy/0.log" Jul 29 19:18:22.632476 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.632451 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-24z2z_38f8b54c-882f-4036-be37-7f42cdbc0f5d/init-textfile/0.log" Jul 29 19:18:22.999011 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:22.998979 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/prometheus/0.log" Jul 29 19:18:23.037217 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.037183 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/config-reloader/0.log" Jul 29 19:18:23.060471 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.060449 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/thanos-sidecar/0.log" Jul 29 19:18:23.091720 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.091697 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/kube-rbac-proxy-web/0.log" Jul 29 19:18:23.129370 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.129267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/kube-rbac-proxy/0.log" Jul 29 19:18:23.163702 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.163674 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/kube-rbac-proxy-thanos/0.log" Jul 29 19:18:23.197872 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:23.197849 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0ac9c2e4-edf6-4672-9e9d-ccd92677652e/init-config-reloader/0.log" Jul 29 19:18:25.766322 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:25.766293 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59c47d49bf-r79rd_c4aa345e-90d8-4231-a250-8487a47e5e57/console/0.log" Jul 29 19:18:26.056636 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.056606 2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv"] Jul 29 19:18:26.056969 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.056917 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="storage-initializer" Jul 29 19:18:26.056969 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.056931 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="storage-initializer" Jul 29 19:18:26.056969 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.056954 2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="main" Jul 29 19:18:26.056969 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.056962 2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="main" Jul 29 19:18:26.057200 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.057016 2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b353b3a-f999-4f7c-a448-0f8aa9367213" containerName="main" Jul 29 19:18:26.059614 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.059593 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.062319 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.062298 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xjw8s\"/\"kube-root-ca.crt\"" Jul 29 19:18:26.063250 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.063230 2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xjw8s\"/\"openshift-service-ca.crt\"" Jul 29 19:18:26.063350 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.063230 2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xjw8s\"/\"default-dockercfg-z2hkg\"" Jul 29 19:18:26.072672 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.072648 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv"] Jul 29 19:18:26.232281 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.232244 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-sys\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.232508 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.232305 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vprxl\" (UniqueName: \"kubernetes.io/projected/ab6ff408-4228-452d-a73c-489690f9f866-kube-api-access-vprxl\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.232508 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.232418 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-lib-modules\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.232508 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.232466 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-podres\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.232508 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.232492 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-proc\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333297 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333208 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-lib-modules\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333297 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-podres\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333297 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333263 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-proc\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333327 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-sys\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333338 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-proc\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333382 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-podres\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333399 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-sys\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333398 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab6ff408-4228-452d-a73c-489690f9f866-lib-modules\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.333606 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.333465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vprxl\" (UniqueName: \"kubernetes.io/projected/ab6ff408-4228-452d-a73c-489690f9f866-kube-api-access-vprxl\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.341902 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.341876 2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vprxl\" (UniqueName: \"kubernetes.io/projected/ab6ff408-4228-452d-a73c-489690f9f866-kube-api-access-vprxl\") pod \"perf-node-gather-daemonset-lpxcv\" (UID: \"ab6ff408-4228-452d-a73c-489690f9f866\") " pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.369223 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.369197 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:26.499921 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:26.499896 2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv"] Jul 29 19:18:26.502467 ip-10-0-129-174 kubenswrapper[2578]: W0729 19:18:26.502443 2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podab6ff408_4228_452d_a73c_489690f9f866.slice/crio-44910335d69d22c1e89106c197e4c98777b31bab6e410d8af6e33aa538111903 WatchSource:0}: Error finding container 44910335d69d22c1e89106c197e4c98777b31bab6e410d8af6e33aa538111903: Status 404 returned error can't find the container with id 44910335d69d22c1e89106c197e4c98777b31bab6e410d8af6e33aa538111903 Jul 29 19:18:27.386205 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.386166 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" event={"ID":"ab6ff408-4228-452d-a73c-489690f9f866","Type":"ContainerStarted","Data":"87b8342d83b6c4b6b0ab732fbea8e4f4a8363e008df9f5dd2ea8222d476c9862"} Jul 29 19:18:27.386683 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.386216 2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" event={"ID":"ab6ff408-4228-452d-a73c-489690f9f866","Type":"ContainerStarted","Data":"44910335d69d22c1e89106c197e4c98777b31bab6e410d8af6e33aa538111903"} Jul 29 19:18:27.386683 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.386274 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:27.404010 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.403953 2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" podStartSLOduration=1.403930564 podStartE2EDuration="1.403930564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:18:27.402938353 +0000 UTC m=+2624.618103384" watchObservedRunningTime="2026-07-29 19:18:27.403930564 +0000 UTC m=+2624.619095592" Jul 29 19:18:27.624746 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.624715 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztqnv_c77ec832-fc29-45f0-9ad7-19ce55fe5600/dns/0.log" Jul 29 19:18:27.649443 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.649376 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ztqnv_c77ec832-fc29-45f0-9ad7-19ce55fe5600/kube-rbac-proxy/0.log" Jul 29 19:18:27.728334 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:27.728300 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wpmjn_5730df58-4eca-443a-8bbf-31ffe7667927/dns-node-resolver/0.log" Jul 29 19:18:28.273990 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:28.273954 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-94cdb7f8d-gqpx8_ff8efbe7-3e36-4ecb-a9e3-b518740ca2ed/registry/0.log" Jul 29 19:18:28.382025 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:28.381994 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tr8pb_703e1173-0804-4069-8845-99e75777cd3b/node-ca/0.log" Jul 29 19:18:29.387572 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:29.387528 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9fbfc_12c5b6c9-adbc-408b-9956-5122d977d12f/discovery/0.log" Jul 29 19:18:30.028777 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:30.028732 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-p2hrm_04ae797e-8c19-4639-b2b9-79d792c53009/serve-healthcheck-canary/0.log" Jul 29 19:18:30.533464 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:30.533434 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67sk6_e41bc761-fa0b-4f31-889f-154be250fbbb/kube-rbac-proxy/0.log" Jul 29 19:18:30.559539 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:30.559512 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67sk6_e41bc761-fa0b-4f31-889f-154be250fbbb/exporter/0.log" Jul 29 19:18:30.589500 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:30.589465 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-67sk6_e41bc761-fa0b-4f31-889f-154be250fbbb/extractor/0.log" Jul 29 19:18:33.400436 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:33.400406 2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xjw8s/perf-node-gather-daemonset-lpxcv" Jul 29 19:18:33.513707 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:33.513676 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7698b7b56c-xwcxp_c2995352-1f2d-42e8-8db1-2925ac9f658a/manager/0.log" Jul 29 19:18:37.087630 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:37.087598 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9fbfc_12c5b6c9-adbc-408b-9956-5122d977d12f/discovery/0.log" Jul 29 19:18:38.242668 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:38.242631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-9fbfc_12c5b6c9-adbc-408b-9956-5122d977d12f/discovery/0.log" Jul 29 19:18:41.711752 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.711719 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/kube-multus-additional-cni-plugins/0.log" Jul 29 19:18:41.737490 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.737461 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/egress-router-binary-copy/0.log" Jul 29 19:18:41.766235 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.766206 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/cni-plugins/0.log" Jul 29 19:18:41.791298 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.791267 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/bond-cni-plugin/0.log" Jul 29 19:18:41.816277 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.816242 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/routeoverride-cni/0.log" Jul 29 19:18:41.844513 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.844480 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/whereabouts-cni-bincopy/0.log" Jul 29 19:18:41.869659 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.869631 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t68xd_cf0d1aa8-2165-473b-a77d-25e2af93723a/whereabouts-cni/0.log" Jul 29 19:18:41.977117 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:41.977040 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wdk2w_afd0e76f-9dd0-429b-95ad-214ee9962d1d/kube-multus/0.log" Jul 29 19:18:42.111132 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:42.111103 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q4h9l_28edf3c8-c9a6-4b2f-a826-7644923b3345/network-metrics-daemon/0.log" Jul 29 19:18:42.135250 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:42.135224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-q4h9l_28edf3c8-c9a6-4b2f-a826-7644923b3345/kube-rbac-proxy/0.log" Jul 29 19:18:43.277523 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.277495 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-controller/0.log" Jul 29 19:18:43.299754 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.299727 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/0.log" Jul 29 19:18:43.299894 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.299761 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovn-acl-logging/1.log" Jul 29 19:18:43.318693 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.318668 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/kube-rbac-proxy-node/0.log" Jul 29 19:18:43.342652 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.342611 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/kube-rbac-proxy-ovn-metrics/0.log" Jul 29 19:18:43.379784 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.379751 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/northd/0.log" Jul 29 19:18:43.419563 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.419514 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/nbdb/0.log" Jul 29 19:18:43.444455 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.444430 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/sbdb/0.log" Jul 29 19:18:43.494354 ip-10-0-129-174 kubenswrapper[2578]: I0729 19:18:43.494324 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jrhmw_e637449e-59bf-41ce-a871-63f3c81263fe/ovnkube-controller/0.log"