Jul 29 21:16:24.181268 ip-10-0-141-233 systemd[1]: Starting Kubernetes Kubelet... Jul 29 21:16:24.627050 ip-10-0-141-233 kubenswrapper[2527]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 21:16:24.627050 ip-10-0-141-233 kubenswrapper[2527]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 21:16:24.627050 ip-10-0-141-233 kubenswrapper[2527]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 21:16:24.627050 ip-10-0-141-233 kubenswrapper[2527]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 21:16:24.627050 ip-10-0-141-233 kubenswrapper[2527]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 21:16:24.629871 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.629783 2527 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 21:16:24.636387 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636370 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.636387 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636387 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636391 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636394 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636397 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636401 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636404 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636407 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636409 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636412 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636415 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636418 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636421 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636436 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636439 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636442 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636445 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636447 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636450 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636453 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.636460 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636457 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636461 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636464 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636467 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636470 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636473 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636476 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636479 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636481 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636485 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636488 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636490 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636493 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636495 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636497 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636500 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636502 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636505 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636508 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.636904 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636510 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636512 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636515 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636517 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636519 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636522 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636525 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636528 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636530 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636533 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636535 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636538 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636540 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636543 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636545 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636548 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636550 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636553 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636556 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636559 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636561 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.637371 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636564 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636585 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636588 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636592 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636597 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636601 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636604 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636607 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636610 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636612 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636615 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636626 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636630 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636633 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636636 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636638 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636641 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636643 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636646 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.637898 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636649 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636653 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636656 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636659 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636662 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636664 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636667 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636669 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636673 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636675 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636678 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636681 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636683 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636685 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636688 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636691 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636693 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636696 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636698 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.638418 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.636700 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637429 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637436 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637439 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637442 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637445 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637447 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637449 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637453 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637455 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637459 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637464 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637467 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637471 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637476 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637479 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637481 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637485 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637487 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.638899 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637490 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637492 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637495 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637497 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637500 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637502 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637505 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637507 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637510 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637513 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637515 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637518 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637520 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637523 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637525 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637528 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637530 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637533 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637535 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637537 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.639419 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637540 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637542 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637545 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637547 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637550 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637553 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637556 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637558 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637562 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637564 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637567 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637569 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637571 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637574 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637577 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637579 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637583 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637585 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637587 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637590 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.639901 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637592 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637595 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637597 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637599 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637602 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637604 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637606 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637609 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637611 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637614 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637616 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637619 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637621 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637624 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637626 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637629 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637631 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637635 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637637 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.640441 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637639 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637642 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637645 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637648 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637650 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637653 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637657 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637659 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637662 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637666 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637669 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637671 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637674 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637676 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637679 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637681 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637684 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637686 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637690 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637694 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.640914 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637696 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.637699 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638098 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638104 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638106 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638109 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638112 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638114 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638117 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638119 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638122 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638125 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638128 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638131 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638134 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638137 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638139 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638142 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638145 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.641426 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638148 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638150 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638153 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638156 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638158 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638161 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638164 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638166 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638169 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638171 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638174 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638176 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638179 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638181 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638184 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638186 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638190 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638194 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638197 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638200 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.641889 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638202 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638205 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638208 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638211 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638214 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638216 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638227 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638230 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638237 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638240 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638243 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638245 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638248 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638252 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638257 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638259 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638262 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638264 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638267 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638269 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.642407 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638272 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638274 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638277 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638279 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638282 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638284 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638287 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638289 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638292 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638295 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638298 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638300 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638303 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638305 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638308 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638311 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638313 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638316 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638319 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638321 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.642896 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638323 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638328 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638330 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638333 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638336 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638338 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638340 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638343 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638345 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638348 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638350 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638353 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638355 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638358 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638360 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638363 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638365 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638368 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638370 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.643384 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638374 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638377 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.638380 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638451 2527 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638458 2527 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638465 2527 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638469 2527 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638473 2527 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638477 2527 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638481 2527 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638485 2527 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638489 2527 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638492 2527 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638495 2527 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638499 2527 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638502 2527 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638506 2527 flags.go:64] FLAG: --cgroup-root="" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638508 2527 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638511 2527 flags.go:64] FLAG: --client-ca-file="" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638514 2527 flags.go:64] FLAG: --cloud-provider="external" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638517 2527 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638521 2527 flags.go:64] FLAG: --cluster-domain="" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638524 2527 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638527 2527 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 21:16:24.643868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638530 2527 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638533 2527 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638537 2527 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638540 2527 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638543 2527 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638547 2527 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638550 2527 flags.go:64] FLAG: --contention-profiling="false" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638553 2527 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638556 2527 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638559 2527 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638562 2527 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638566 2527 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638569 2527 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638572 2527 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638575 2527 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638578 2527 flags.go:64] FLAG: --enable-server="true" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638582 2527 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638586 2527 flags.go:64] FLAG: --event-burst="100" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638589 2527 flags.go:64] FLAG: --event-qps="50" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638592 2527 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638595 2527 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638600 2527 flags.go:64] FLAG: --eviction-hard="" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638608 2527 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638611 2527 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638615 2527 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 21:16:24.644456 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638618 2527 flags.go:64] FLAG: --eviction-soft="" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638621 2527 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638623 2527 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638626 2527 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638629 2527 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638632 2527 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638634 2527 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638637 2527 flags.go:64] FLAG: --feature-gates="" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638640 2527 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638643 2527 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638646 2527 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638649 2527 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638652 2527 flags.go:64] FLAG: --healthz-port="10248" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638655 2527 flags.go:64] FLAG: --help="false" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638658 2527 flags.go:64] FLAG: --hostname-override="ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638661 2527 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638664 2527 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638667 2527 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638670 2527 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638673 2527 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638676 2527 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638679 2527 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638681 2527 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 21:16:24.645066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638684 2527 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638686 2527 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638689 2527 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638692 2527 flags.go:64] FLAG: --kube-reserved="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638695 2527 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638699 2527 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638702 2527 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638705 2527 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638708 2527 flags.go:64] FLAG: --lock-file="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638711 2527 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638714 2527 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638717 2527 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638723 2527 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638726 2527 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638728 2527 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638731 2527 flags.go:64] FLAG: --logging-format="text" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638734 2527 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638737 2527 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638742 2527 flags.go:64] FLAG: --manifest-url="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638744 2527 flags.go:64] FLAG: --manifest-url-header="" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638748 2527 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638752 2527 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638756 2527 flags.go:64] FLAG: --max-pods="110" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638759 2527 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638762 2527 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 21:16:24.645686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638765 2527 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638771 2527 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638774 2527 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638777 2527 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638780 2527 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638787 2527 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638790 2527 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638793 2527 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638796 2527 flags.go:64] FLAG: --pod-cidr="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638798 2527 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638804 2527 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638806 2527 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638809 2527 flags.go:64] FLAG: --pods-per-core="0" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638813 2527 flags.go:64] FLAG: --port="10250" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638816 2527 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638819 2527 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0da6fb84b77f058b5" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638822 2527 flags.go:64] FLAG: --qos-reserved="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638826 2527 flags.go:64] FLAG: --read-only-port="10255" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638829 2527 flags.go:64] FLAG: --register-node="true" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638832 2527 flags.go:64] FLAG: --register-with-taints="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638835 2527 flags.go:64] FLAG: --registry-burst="10" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638838 2527 flags.go:64] FLAG: --registry-qps="5" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638841 2527 flags.go:64] FLAG: --reserved-cpus="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638843 2527 flags.go:64] FLAG: --reserved-memory="" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638847 2527 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 21:16:24.646379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638852 2527 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638855 2527 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638857 2527 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638860 2527 flags.go:64] FLAG: --runonce="false" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638863 2527 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638866 2527 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638869 2527 flags.go:64] FLAG: --seccomp-default="false" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638871 2527 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638874 2527 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638877 2527 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638880 2527 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638883 2527 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638885 2527 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638888 2527 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638890 2527 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638893 2527 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638896 2527 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638899 2527 flags.go:64] FLAG: --system-cgroups="" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638901 2527 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638907 2527 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638913 2527 flags.go:64] FLAG: --tls-cert-file="" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638916 2527 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638920 2527 flags.go:64] FLAG: --tls-min-version="" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638938 2527 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638941 2527 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 21:16:24.647051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638944 2527 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638947 2527 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638950 2527 flags.go:64] FLAG: --v="2" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638954 2527 flags.go:64] FLAG: --version="false" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638958 2527 flags.go:64] FLAG: --vmodule="" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638962 2527 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.638965 2527 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639084 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639089 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639092 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639095 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639098 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639101 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639104 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639106 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639109 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639112 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639114 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639116 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639119 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639122 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.648021 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639124 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639126 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639129 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639131 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639133 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639136 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639139 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639142 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639144 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639148 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639151 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639154 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639157 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639160 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639163 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639166 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639168 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639171 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639175 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.648543 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639177 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639180 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639183 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639185 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639187 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639190 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639193 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639195 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639198 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639200 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639203 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639205 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639207 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639210 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639213 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639216 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639219 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639222 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639225 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639228 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.649010 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639231 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639233 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639236 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639238 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639241 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639243 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639246 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639249 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639252 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639254 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639257 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639260 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639263 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639265 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639268 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639270 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639272 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639275 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639277 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.649504 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639280 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639282 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639285 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639287 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639290 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639292 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639295 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639297 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639299 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639302 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639304 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639307 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639311 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639314 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639316 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639319 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639321 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639323 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639327 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639329 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.650136 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639332 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639335 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639337 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639340 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639343 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639346 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.639348 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.640051 2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.649290 2527 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.649306 2527 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.649326 2527 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.649332 2527 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649390 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649395 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649398 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.650637 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649401 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649405 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649409 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649412 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649414 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649417 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649420 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649422 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649425 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649428 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649431 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649434 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649437 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649439 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649442 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649444 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649447 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649449 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649452 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.651029 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649454 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649457 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649460 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649462 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649467 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649470 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649473 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649476 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649478 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649481 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649484 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649486 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649489 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649491 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649494 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649496 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649499 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649501 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649504 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649507 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.651505 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649509 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649512 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649515 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649519 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649522 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649525 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649528 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649532 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649535 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649537 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649540 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649542 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649545 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649548 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649551 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649554 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649556 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649558 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649561 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.651997 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649564 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649566 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649569 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649571 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649574 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649576 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649579 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649581 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649583 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649586 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649588 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649591 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649593 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649595 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649598 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649600 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649603 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649605 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649608 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649611 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.652452 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649613 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649616 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649619 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649621 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649624 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649626 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649628 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649631 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649633 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649635 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649638 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649640 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649643 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649645 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649648 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649650 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649653 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.652959 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649655 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.649660 2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649773 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649778 2527 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649781 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649784 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649786 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649789 2527 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649791 2527 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649794 2527 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649797 2527 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649799 2527 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649802 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649806 2527 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649811 2527 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 21:16:24.653380 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649814 2527 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649817 2527 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649820 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649823 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649825 2527 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649827 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649830 2527 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649832 2527 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649835 2527 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649837 2527 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649840 2527 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649842 2527 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649845 2527 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649847 2527 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649850 2527 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649852 2527 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649854 2527 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649857 2527 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649859 2527 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649861 2527 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 21:16:24.653791 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649864 2527 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649866 2527 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649869 2527 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649871 2527 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649874 2527 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649876 2527 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649878 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649881 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649884 2527 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649887 2527 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649890 2527 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649893 2527 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649899 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649901 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649904 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649907 2527 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649909 2527 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649911 2527 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649914 2527 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 21:16:24.654360 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649916 2527 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649919 2527 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649922 2527 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649938 2527 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649941 2527 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649943 2527 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649946 2527 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649951 2527 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649954 2527 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649957 2527 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649959 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649963 2527 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649965 2527 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649968 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649971 2527 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649973 2527 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649976 2527 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649978 2527 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649981 2527 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 21:16:24.654830 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649983 2527 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649986 2527 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649988 2527 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649991 2527 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649994 2527 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649996 2527 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.649999 2527 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650002 2527 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650004 2527 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650007 2527 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650009 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650012 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650015 2527 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650018 2527 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650020 2527 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650023 2527 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650025 2527 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650028 2527 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650030 2527 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 21:16:24.655314 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650033 2527 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650035 2527 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650038 2527 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650040 2527 feature_gate.go:328] unrecognized feature gate: Example Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650042 2527 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650045 2527 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650048 2527 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650050 2527 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:24.650052 2527 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.650057 2527 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.650197 2527 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.652787 2527 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.654460 2527 server.go:1019] "Starting client certificate rotation" Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.654568 2527 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 21:16:24.655764 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.654618 2527 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 21:16:24.657313 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.657299 2527 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 21:16:24.661239 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.661217 2527 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 21:16:24.673326 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.673309 2527 log.go:25] "Validated CRI v1 runtime API" Jul 29 21:16:24.676004 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.675987 2527 log.go:25] "Validated CRI v1 image API" Jul 29 21:16:24.678481 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.678459 2527 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 21:16:24.681226 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.681208 2527 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a423fcb7-c519-49c6-97b5-77c85b47e0e0:/dev/nvme0n1p4 a9d19550-6417-44aa-86b6-f974aaaf589e:/dev/nvme0n1p3] Jul 29 21:16:24.681278 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.681227 2527 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 21:16:24.688141 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.688019 2527 manager.go:216] Machine: {Timestamp:2026-07-29 21:16:24.686297609 +0000 UTC m=+0.376729418 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100209 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e075f38a7b23bc481e94a13bb1103 SystemUUID:ec2e075f-38a7-b23b-c481-e94a13bb1103 BootID:2f249e48-104d-4ac5-9df6-874f61a7806a Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:2e:4b:ff:ed:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:2e:4b:ff:ed:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:96:1c:20:71:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 21:16:24.688141 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.688128 2527 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 21:16:24.688275 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.688215 2527 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 21:16:24.691751 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.691719 2527 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 21:16:24.691887 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.691755 2527 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 21:16:24.692725 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.692715 2527 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 21:16:24.692758 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.692727 2527 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 21:16:24.692758 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.692734 2527 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 21:16:24.693562 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.693553 2527 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 21:16:24.694853 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.694842 2527 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 21:16:24.697780 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697765 2527 state_mem.go:36] "Initialized new in-memory state store" Jul 29 21:16:24.697896 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697886 2527 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 21:16:24.697947 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697942 2527 kubelet.go:486] "Attempting to sync node with API server" Jul 29 21:16:24.697990 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697957 2527 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 21:16:24.697990 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697981 2527 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 21:16:24.698051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.697995 2527 kubelet.go:398] "Adding apiserver pod source" Jul 29 21:16:24.698051 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.698007 2527 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 21:16:24.698453 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.698440 2527 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 21:16:24.698488 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.698460 2527 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 21:16:24.699569 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.699549 2527 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 21:16:24.700725 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.700709 2527 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 21:16:24.703078 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.703065 2527 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 21:16:24.703123 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.703095 2527 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 21:16:24.705147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705133 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705153 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705160 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705165 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705173 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705178 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 21:16:24.705184 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705186 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 21:16:24.705341 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705191 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 21:16:24.705341 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705197 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 21:16:24.705341 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705203 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 21:16:24.705341 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705211 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 21:16:24.705341 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.705216 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 21:16:24.706138 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.706129 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 21:16:24.706138 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.706138 2527 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 21:16:24.709556 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.709541 2527 server.go:1268] "Started kubelet" Jul 29 21:16:24.709870 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.709850 2527 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 21:16:24.709996 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.709807 2527 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 21:16:24.709996 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.709908 2527 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 21:16:24.710150 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.710018 2527 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 21:16:24.710570 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.710543 2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 21:16:24.710570 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.710544 2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 21:16:24.710846 ip-10-0-141-233 systemd[1]: Started Kubernetes Kubelet. Jul 29 21:16:24.712247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.712231 2527 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 21:16:24.712709 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.712690 2527 server.go:310] "Adding debug handlers to kubelet server" Jul 29 21:16:24.715383 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.715365 2527 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 21:16:24.717562 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.716403 2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-233.ec2.internal.18c6df43d3e0d645 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-233.ec2.internal,UID:ip-10-0-141-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-233.ec2.internal,},FirstTimestamp:2026-07-29 21:16:24.709510725 +0000 UTC m=+0.399942537,LastTimestamp:2026-07-29 21:16:24.709510725 +0000 UTC m=+0.399942537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-233.ec2.internal,}" Jul 29 21:16:24.722345 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.722326 2527 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 21:16:24.722345 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.722333 2527 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 21:16:24.722566 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.722550 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:24.723102 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723077 2527 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 21:16:24.723197 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723112 2527 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 21:16:24.723197 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723081 2527 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 21:16:24.723291 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723279 2527 reconstruct.go:97] "Volume reconstruction finished" Jul 29 21:16:24.723336 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723313 2527 reconciler.go:29] "Reconciler: start to sync state" Jul 29 21:16:24.723437 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723420 2527 factory.go:55] Registering systemd factory Jul 29 21:16:24.723437 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.723436 2527 factory.go:223] Registration of the systemd container factory successfully Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.725366 2527 factory.go:153] Registering CRI-O factory Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.725371 2527 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.725381 2527 factory.go:223] Registration of the crio container factory successfully Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.725471 2527 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.725500 2527 factory.go:103] Registering Raw factory Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.725491 2527 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 21:16:24.725571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.725521 2527 manager.go:1196] Started watching for new ooms in manager Jul 29 21:16:24.726251 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.726237 2527 manager.go:349] Starting recovery of all containers Jul 29 21:16:24.737504 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.737305 2527 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 21:16:24.738308 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.738291 2527 manager.go:354] Recovery completed Jul 29 21:16:24.743231 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.743216 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.745137 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745116 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.745217 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745155 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.745217 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745172 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.745708 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745694 2527 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 21:16:24.745770 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745709 2527 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 21:16:24.745770 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745728 2527 state_mem.go:36] "Initialized new in-memory state store" Jul 29 21:16:24.745770 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.745730 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.746497 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.746482 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.746550 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.746513 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.746550 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.746528 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.748553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.748508 2527 policy_none.go:49] "None policy: Start" Jul 29 21:16:24.748553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.748524 2527 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 21:16:24.748553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.748535 2527 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 21:16:24.750551 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.750539 2527 policy_none.go:47] "Start" Jul 29 21:16:24.755689 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.755595 2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-233.ec2.internal.18c6df43d60075f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-233.ec2.internal,UID:ip-10-0-141-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-233.ec2.internal,},FirstTimestamp:2026-07-29 21:16:24.745137651 +0000 UTC m=+0.435569461,LastTimestamp:2026-07-29 21:16:24.745137651 +0000 UTC m=+0.435569461,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-233.ec2.internal,}" Jul 29 21:16:24.766974 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.766889 2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-233.ec2.internal.18c6df43d600dc1c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-233.ec2.internal,UID:ip-10-0-141-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-233.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-233.ec2.internal,},FirstTimestamp:2026-07-29 21:16:24.745163804 +0000 UTC m=+0.435595616,LastTimestamp:2026-07-29 21:16:24.745163804 +0000 UTC m=+0.435595616,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-233.ec2.internal,}" Jul 29 21:16:24.778598 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.778538 2527 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-233.ec2.internal.18c6df43d60112d9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-233.ec2.internal,UID:ip-10-0-141-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-233.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-233.ec2.internal,},FirstTimestamp:2026-07-29 21:16:24.745177817 +0000 UTC m=+0.435609628,LastTimestamp:2026-07-29 21:16:24.745177817 +0000 UTC m=+0.435609628,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-233.ec2.internal,}" Jul 29 21:16:24.780578 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.780559 2527 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwwbq" Jul 29 21:16:24.790229 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.790203 2527 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-zwwbq" Jul 29 21:16:24.790394 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.789224 2527 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-141-233.ec2.internal.18c6df43d60075f3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-233.ec2.internal.18c6df43d60075f3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-233.ec2.internal,UID:ip-10-0-141-233.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-233.ec2.internal,},FirstTimestamp:2026-07-29 21:16:24.745137651 +0000 UTC m=+0.435569461,LastTimestamp:2026-07-29 21:16:24.746498007 +0000 UTC m=+0.436929822,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-233.ec2.internal,}" Jul 29 21:16:24.792150 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792136 2527 manager.go:337] "Starting Device Plugin manager" Jul 29 21:16:24.792234 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.792180 2527 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 21:16:24.792234 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792191 2527 server.go:87] "Starting device plugin registration server" Jul 29 21:16:24.792451 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792439 2527 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 21:16:24.792511 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792454 2527 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 21:16:24.792627 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792606 2527 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 21:16:24.792748 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792726 2527 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 21:16:24.792748 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.792734 2527 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 21:16:24.793319 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.793295 2527 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 21:16:24.793469 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.793450 2527 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:24.846560 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.846528 2527 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 21:16:24.848728 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.846574 2527 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 21:16:24.848728 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.846597 2527 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 21:16:24.848728 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.846629 2527 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 21:16:24.849288 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.849273 2527 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 21:16:24.893267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.893197 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.894206 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.894188 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.894346 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.894229 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.894346 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.894244 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.894346 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.894272 2527 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.907250 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.907230 2527 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.907317 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.907258 2527 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-233.ec2.internal\": node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:24.927805 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.927773 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:24.946721 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.946691 2527 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal"] Jul 29 21:16:24.946858 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.946767 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.948314 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.948286 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.948421 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.948326 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.948421 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.948339 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.950549 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.950537 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.950685 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.950670 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.950733 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.950699 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.951269 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.951251 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.951373 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.951283 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.951373 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.951298 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.952470 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.952451 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.952555 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.952481 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.952555 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.952491 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.954702 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.954685 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.954792 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.954712 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:24.955423 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.955401 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:24.955519 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.955438 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:24.955519 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:24.955452 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:24.982901 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.982874 2527 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-233.ec2.internal\" not found" node="ip-10-0-141-233.ec2.internal" Jul 29 21:16:24.987281 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:24.987263 2527 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-233.ec2.internal\" not found" node="ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.034154 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.034120 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.134908 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.134869 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.135113 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.135004 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.135113 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.135035 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.135113 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.135058 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b454936a2c3aaec87ddb9b7f44eca43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-233.ec2.internal\" (UID: \"9b454936a2c3aaec87ddb9b7f44eca43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235597 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.235527 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.235597 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235569 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235602 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235618 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b454936a2c3aaec87ddb9b7f44eca43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-233.ec2.internal\" (UID: \"9b454936a2c3aaec87ddb9b7f44eca43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235655 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b454936a2c3aaec87ddb9b7f44eca43-config\") pod \"kube-apiserver-proxy-ip-10-0-141-233.ec2.internal\" (UID: \"9b454936a2c3aaec87ddb9b7f44eca43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235657 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.235730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.235669 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6a7dffde4427b22d8c34d040f4ff0c1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal\" (UID: \"b6a7dffde4427b22d8c34d040f4ff0c1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.285760 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.285722 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.291092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.291058 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.336606 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.336575 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.437049 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.437007 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.537424 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.537360 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.637752 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.637724 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.653916 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.653886 2527 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 21:16:25.654096 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.654073 2527 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 21:16:25.738419 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.738388 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.796876 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.796787 2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 21:11:25 +0000 UTC" deadline="2028-03-28 18:02:14.304103298 +0000 UTC" Jul 29 21:16:25.796876 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.796834 2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14588h45m48.507272439s" Jul 29 21:16:25.839200 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:25.839168 2527 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-233.ec2.internal\" not found" Jul 29 21:16:25.871008 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.870977 2527 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 21:16:25.874917 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.874898 2527 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 21:16:25.876196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.876179 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientMemory" Jul 29 21:16:25.876271 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.876208 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 21:16:25.876271 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.876218 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeHasSufficientPID" Jul 29 21:16:25.881306 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.881286 2527 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 21:16:25.923474 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.923451 2527 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.930404 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:25.930370 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a7dffde4427b22d8c34d040f4ff0c1.slice/crio-c17cbe93b11376a9c29e4c034ea83b056df227c071e86291c7b1c5eda2dca978 WatchSource:0}: Error finding container c17cbe93b11376a9c29e4c034ea83b056df227c071e86291c7b1c5eda2dca978: Status 404 returned error can't find the container with id c17cbe93b11376a9c29e4c034ea83b056df227c071e86291c7b1c5eda2dca978 Jul 29 21:16:25.930877 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:25.930857 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b454936a2c3aaec87ddb9b7f44eca43.slice/crio-0a3861a780dcfd8001ab4a91636014b81c67dc3f22d6d333d483f381e7ed484c WatchSource:0}: Error finding container 0a3861a780dcfd8001ab4a91636014b81c67dc3f22d6d333d483f381e7ed484c: Status 404 returned error can't find the container with id 0a3861a780dcfd8001ab4a91636014b81c67dc3f22d6d333d483f381e7ed484c Jul 29 21:16:25.935321 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.935305 2527 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 21:16:25.935382 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.935335 2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:16:25.936349 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.936335 2527 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" Jul 29 21:16:25.946619 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:25.946603 2527 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 21:16:26.204650 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.204563 2527 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 21:16:26.698706 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.698585 2527 apiserver.go:52] "Watching apiserver" Jul 29 21:16:26.706198 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.706166 2527 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 21:16:26.706782 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.706752 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kv7sc","kube-system/konnectivity-agent-wq45j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw","openshift-dns/node-resolver-fxx4w","openshift-multus/multus-additional-cni-plugins-t5hk2","openshift-multus/multus-fnxnn","openshift-multus/network-metrics-daemon-9qszv","kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal","openshift-cluster-node-tuning-operator/tuned-qh9jq","openshift-image-registry/node-ca-ztx7p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal","openshift-network-diagnostics/network-check-target-zhtjs","openshift-network-operator/iptables-alerter-l8469","openshift-ovn-kubernetes/ovnkube-node-djvst"] Jul 29 21:16:26.709845 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.709824 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.710014 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.709908 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:26.712909 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.712159 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.716139 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.715411 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.716822 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.716568 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.716822 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.716678 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 21:16:26.716822 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.716697 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-x6v47\"" Jul 29 21:16:26.716822 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.716569 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.717793 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.717755 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.717941 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.717881 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 21:16:26.718144 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.718114 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9dlx5\"" Jul 29 21:16:26.718224 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.718144 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.718393 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.718376 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.720686 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.720664 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.721326 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.721304 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-452fh\"" Jul 29 21:16:26.721421 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.721404 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.721497 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.721472 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.722933 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.722910 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 21:16:26.723206 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.723184 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 21:16:26.725710 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.725690 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lt4vl\"" Jul 29 21:16:26.726902 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.726254 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.726902 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.726527 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 21:16:26.726902 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.726806 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.729035 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.728230 2527 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 21:16:26.729132 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.729108 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.729328 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.729298 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-tdvf6\"" Jul 29 21:16:26.729522 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.729505 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 21:16:26.731223 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.731001 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.733447 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.733393 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.734719 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.733909 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.734719 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.734142 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:26.734719 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.734252 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.735554 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.735533 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-l4np9\"" Jul 29 21:16:26.737532 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.737512 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.741101 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.740632 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.741101 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.740738 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 21:16:26.742174 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.741537 2527 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 21:16:26.742174 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.741778 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 21:16:26.742174 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.741980 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hk8sh\"" Jul 29 21:16:26.743082 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.743057 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 21:16:26.743482 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.743451 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2l4n6\"" Jul 29 21:16:26.743556 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.743488 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.744204 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.744184 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.744834 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.744804 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-os-release\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745011 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.744993 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hxd6\" (UniqueName: \"kubernetes.io/projected/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-kube-api-access-6hxd6\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.745247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745126 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514a6c7e-c678-4089-8371-2f50ec8bd987-tmp-dir\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.745247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745158 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.745247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745184 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-socket-dir-parent\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745237 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-bin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745273 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-etc-kubernetes\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745300 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwhr\" (UniqueName: \"kubernetes.io/projected/946d043a-d4b9-42bc-87e5-41db6c5d7fde-kube-api-access-dhwhr\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745324 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-kubelet\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745348 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745371 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745394 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-system-cni-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745419 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-k8s-cni-cncf-io\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745453 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-multus\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745491 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-conf-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745516 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-daemon-config\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745542 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745566 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-host-slash\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745609 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-device-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745641 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745669 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cnibin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745722 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-netns\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745757 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745789 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-hostroot\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745829 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cni-binary-copy\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745875 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.745952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745901 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.745970 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mszvs\" (UniqueName: \"kubernetes.io/projected/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-kube-api-access-mszvs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746003 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746029 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnvz\" (UniqueName: \"kubernetes.io/projected/514a6c7e-c678-4089-8371-2f50ec8bd987-kube-api-access-gnnvz\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746063 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746087 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-multus-certs\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746128 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzpl2\" (UniqueName: \"kubernetes.io/projected/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-kube-api-access-zzpl2\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746152 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746175 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-system-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746208 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-iptables-alerter-script\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746248 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456tr\" (UniqueName: \"kubernetes.io/projected/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kube-api-access-456tr\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746305 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/514a6c7e-c678-4089-8371-2f50ec8bd987-hosts-file\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746330 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cnibin\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.746618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.746365 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-os-release\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.749690 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.749669 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.749912 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.749897 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 21:16:26.750172 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.750143 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 21:16:26.750247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.750232 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 21:16:26.750449 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.750433 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 21:16:26.750814 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.750784 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 21:16:26.750998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.750983 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 21:16:26.751310 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.751292 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8h4d5\"" Jul 29 21:16:26.751556 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.751539 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:26.751634 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.751611 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:26.770131 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.770106 2527 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdx8x" Jul 29 21:16:26.783700 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.783668 2527 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jdx8x" Jul 29 21:16:26.825419 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.825393 2527 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 21:16:26.847132 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847096 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-dbus\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.847132 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847141 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31b11f5-cefa-4201-ae2b-b088d025948a-konnectivity-ca\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847179 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847207 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-multus\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847232 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-conf-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847255 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-daemon-config\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847275 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-sys\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847292 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-etc-tuned\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847326 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-bin\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847341 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-netd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847376 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847364 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847385 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-device-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847400 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847427 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cnibin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847455 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-systemd\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847481 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847506 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-config\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847540 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31b11f5-cefa-4201-ae2b-b088d025948a-agent-certs\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847565 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-kubelet\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847589 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-etc-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847619 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-kubelet-config\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847642 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-ovn\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847686 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-lib-modules\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847713 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.847759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847740 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847768 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mszvs\" (UniqueName: \"kubernetes.io/projected/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-kube-api-access-mszvs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.847795 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848318 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-multus-certs\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848353 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58046276-d71d-4df4-afeb-a5f37f66c360-host\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848379 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848408 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-bin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848442 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-modprobe-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848465 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848498 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848519 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-system-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848537 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-iptables-alerter-script\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848561 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456tr\" (UniqueName: \"kubernetes.io/projected/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kube-api-access-456tr\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848579 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-os-release\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848596 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hxd6\" (UniqueName: \"kubernetes.io/projected/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-kube-api-access-6hxd6\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848612 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514a6c7e-c678-4089-8371-2f50ec8bd987-tmp-dir\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848630 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848648 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-socket-dir-parent\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848676 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-os-release\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848697 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58046276-d71d-4df4-afeb-a5f37f66c360-serviceca\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848718 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-systemd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848735 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-log-socket\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848752 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-kubelet\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848767 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-netns\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.848832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848794 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848819 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-system-cni-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848835 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-k8s-cni-cncf-io\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848855 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-tmp\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848871 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-script-lib\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848901 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-host-slash\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848918 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-netns\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848933 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-var-lib-kubelet\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848954 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8kh7\" (UniqueName: \"kubernetes.io/projected/72c190d6-a300-4510-bcab-fe14c70974cc-kube-api-access-n8kh7\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.848979 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-systemd-units\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849051 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-multus-certs\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849058 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-os-release\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849053 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-k8s-cni-cncf-io\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849112 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-kubelet\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849178 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-node-log\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849217 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66vc\" (UniqueName: \"kubernetes.io/projected/7c58c126-7301-46d4-963d-f5c15a7f8712-kube-api-access-r66vc\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849232 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-host-slash\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.849714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849253 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849263 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-run-netns\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849282 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-hostroot\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849290 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-socket-dir-parent\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849312 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-kubernetes\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849340 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-var-lib-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849371 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cni-binary-copy\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849381 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-multus\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849385 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-etc-selinux\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849395 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-run\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849427 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-host-var-lib-cni-bin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849435 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849456 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849477 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-hostroot\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849485 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-conf-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849480 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-conf\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849454 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-os-release\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.850426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849546 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-system-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849600 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-socket-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849617 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.849655 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-registration-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850235 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cni-binary-copy\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850254 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-binary-copy\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850263 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850256 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-iptables-alerter-script\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850264 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850318 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-cnibin\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850374 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-device-dir\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.850385 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.850457 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.350437872 +0000 UTC m=+3.040869688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850559 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-cni-dir\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850578 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-multus-daemon-config\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850601 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-system-cni-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850607 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/514a6c7e-c678-4089-8371-2f50ec8bd987-tmp-dir\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.851147 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850618 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-sys-fs\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850640 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnvz\" (UniqueName: \"kubernetes.io/projected/514a6c7e-c678-4089-8371-2f50ec8bd987-kube-api-access-gnnvz\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850680 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzpl2\" (UniqueName: \"kubernetes.io/projected/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-kube-api-access-zzpl2\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850723 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850871 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.850921 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxgc5\" (UniqueName: \"kubernetes.io/projected/58046276-d71d-4df4-afeb-a5f37f66c360-kube-api-access-nxgc5\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851545 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-slash\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851580 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysconfig\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851622 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/514a6c7e-c678-4089-8371-2f50ec8bd987-hosts-file\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851650 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cnibin\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851674 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851700 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-env-overrides\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851745 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-etc-kubernetes\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851759 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/946d043a-d4b9-42bc-87e5-41db6c5d7fde-cnibin\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851762 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c58c126-7301-46d4-963d-f5c15a7f8712-ovn-node-metrics-cert\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851809 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/514a6c7e-c678-4089-8371-2f50ec8bd987-hosts-file\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851843 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-etc-kubernetes\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.851937 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851877 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwhr\" (UniqueName: \"kubernetes.io/projected/946d043a-d4b9-42bc-87e5-41db6c5d7fde-kube-api-access-dhwhr\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.852698 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.851923 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-host\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.854665 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.853986 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" event={"ID":"b6a7dffde4427b22d8c34d040f4ff0c1","Type":"ContainerStarted","Data":"c17cbe93b11376a9c29e4c034ea83b056df227c071e86291c7b1c5eda2dca978"} Jul 29 21:16:26.856448 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.856352 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" event={"ID":"9b454936a2c3aaec87ddb9b7f44eca43","Type":"ContainerStarted","Data":"0a3861a780dcfd8001ab4a91636014b81c67dc3f22d6d333d483f381e7ed484c"} Jul 29 21:16:26.861802 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.858941 2527 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 21:16:26.866228 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.866203 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mszvs\" (UniqueName: \"kubernetes.io/projected/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-kube-api-access-mszvs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:26.869316 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.869249 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hxd6\" (UniqueName: \"kubernetes.io/projected/f9ab5b94-4525-4d7f-b9c9-f183e6a75d26-kube-api-access-6hxd6\") pod \"iptables-alerter-l8469\" (UID: \"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26\") " pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:26.870502 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.870475 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnvz\" (UniqueName: \"kubernetes.io/projected/514a6c7e-c678-4089-8371-2f50ec8bd987-kube-api-access-gnnvz\") pod \"node-resolver-fxx4w\" (UID: \"514a6c7e-c678-4089-8371-2f50ec8bd987\") " pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:26.872168 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.872145 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzpl2\" (UniqueName: \"kubernetes.io/projected/b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2-kube-api-access-zzpl2\") pod \"multus-fnxnn\" (UID: \"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2\") " pod="openshift-multus/multus-fnxnn" Jul 29 21:16:26.875738 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.875714 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-456tr\" (UniqueName: \"kubernetes.io/projected/5d5e37d0-5571-4a03-bb99-4cc4b08897f3-kube-api-access-456tr\") pod \"aws-ebs-csi-driver-node-hgvqw\" (UID: \"5d5e37d0-5571-4a03-bb99-4cc4b08897f3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:26.877816 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.877790 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwhr\" (UniqueName: \"kubernetes.io/projected/946d043a-d4b9-42bc-87e5-41db6c5d7fde-kube-api-access-dhwhr\") pod \"multus-additional-cni-plugins-t5hk2\" (UID: \"946d043a-d4b9-42bc-87e5-41db6c5d7fde\") " pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:26.926427 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.926396 2527 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 21:16:26.952374 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952252 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31b11f5-cefa-4201-ae2b-b088d025948a-agent-certs\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.952374 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952307 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-kubelet\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952374 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952335 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-etc-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952374 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952363 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-kubelet-config\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952391 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-ovn\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952417 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-lib-modules\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952459 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58046276-d71d-4df4-afeb-a5f37f66c360-host\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952483 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952517 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-modprobe-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952543 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952577 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58046276-d71d-4df4-afeb-a5f37f66c360-serviceca\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952593 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-systemd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952609 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-log-socket\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.952631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952630 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-netns\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952647 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-tmp\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952662 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-script-lib\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952679 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-var-lib-kubelet\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952694 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8kh7\" (UniqueName: \"kubernetes.io/projected/72c190d6-a300-4510-bcab-fe14c70974cc-kube-api-access-n8kh7\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952709 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-systemd-units\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952724 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-node-log\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952746 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66vc\" (UniqueName: \"kubernetes.io/projected/7c58c126-7301-46d4-963d-f5c15a7f8712-kube-api-access-r66vc\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952771 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-kubernetes\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952788 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-var-lib-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952808 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-run\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952830 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952847 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-conf\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952865 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952888 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxgc5\" (UniqueName: \"kubernetes.io/projected/58046276-d71d-4df4-afeb-a5f37f66c360-kube-api-access-nxgc5\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952917 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-slash\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.952986 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysconfig\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953040 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953067 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-env-overrides\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953094 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c58c126-7301-46d4-963d-f5c15a7f8712-ovn-node-metrics-cert\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953123 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-host\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953146 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-dbus\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953172 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31b11f5-cefa-4201-ae2b-b088d025948a-konnectivity-ca\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953202 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-sys\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953226 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-etc-tuned\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953249 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-bin\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953296 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-netd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953329 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-systemd\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953356 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953390 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-config\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953033 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysconfig\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953497 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-var-lib-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953555 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-var-lib-kubelet\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.953868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953680 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-script-lib\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953885 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-systemd-units\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953921 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-ovnkube-config\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953939 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-node-log\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.953943 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-run\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954020 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-host\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954032 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-kubernetes\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954287 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-netd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954350 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-sys\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954435 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7c58c126-7301-46d4-963d-f5c15a7f8712-env-overrides\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954502 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954655 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-conf\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.954713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.954679 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-cni-bin\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.954758 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.954816 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.454798324 +0000 UTC m=+3.145230143 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955078 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-slash\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955127 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955170 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/58046276-d71d-4df4-afeb-a5f37f66c360-host\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.955291 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955232 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/f31b11f5-cefa-4201-ae2b-b088d025948a-konnectivity-ca\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.955596 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955385 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-systemd\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.955596 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955443 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-kubelet-config\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.955596 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955493 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-kubelet\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955596 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955535 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-etc-openvswitch\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955596 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955574 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-log-socket\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955834 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955613 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-systemd\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955834 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955652 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-run-ovn\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.955834 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955694 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-netns\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955906 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/58046276-d71d-4df4-afeb-a5f37f66c360-serviceca\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.955973 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7c58c126-7301-46d4-963d-f5c15a7f8712-host-run-ovn-kubernetes\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.956119 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-modprobe-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.956149 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b3743918-d652-44d4-a116-68a05a655027-dbus\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.956222 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-etc-sysctl-d\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.956640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.956359 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/72c190d6-a300-4510-bcab-fe14c70974cc-lib-modules\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.956928 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.956777 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-tmp\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.958720 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.958695 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/72c190d6-a300-4510-bcab-fe14c70974cc-etc-tuned\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.958852 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.958797 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/f31b11f5-cefa-4201-ae2b-b088d025948a-agent-certs\") pod \"konnectivity-agent-wq45j\" (UID: \"f31b11f5-cefa-4201-ae2b-b088d025948a\") " pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:26.960560 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.960533 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7c58c126-7301-46d4-963d-f5c15a7f8712-ovn-node-metrics-cert\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:26.961752 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.961730 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:26.961752 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.961759 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:26.961954 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.961773 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:26.961954 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:26.961835 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:27.461817563 +0000 UTC m=+3.152249398 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:26.964779 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.964751 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxgc5\" (UniqueName: \"kubernetes.io/projected/58046276-d71d-4df4-afeb-a5f37f66c360-kube-api-access-nxgc5\") pod \"node-ca-ztx7p\" (UID: \"58046276-d71d-4df4-afeb-a5f37f66c360\") " pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:26.965572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.965552 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8kh7\" (UniqueName: \"kubernetes.io/projected/72c190d6-a300-4510-bcab-fe14c70974cc-kube-api-access-n8kh7\") pod \"tuned-qh9jq\" (UID: \"72c190d6-a300-4510-bcab-fe14c70974cc\") " pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:26.965915 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:26.965890 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66vc\" (UniqueName: \"kubernetes.io/projected/7c58c126-7301-46d4-963d-f5c15a7f8712-kube-api-access-r66vc\") pod \"ovnkube-node-djvst\" (UID: \"7c58c126-7301-46d4-963d-f5c15a7f8712\") " pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:27.038920 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.038857 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l8469" Jul 29 21:16:27.041025 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.040996 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxx4w" Jul 29 21:16:27.054164 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.054138 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" Jul 29 21:16:27.064437 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.064392 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" Jul 29 21:16:27.085689 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.085656 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-fnxnn" Jul 29 21:16:27.093787 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.093756 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" Jul 29 21:16:27.095916 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.095888 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:27.107072 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.107046 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ztx7p" Jul 29 21:16:27.113137 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.113113 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:27.357605 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.357513 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:27.357769 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.357679 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:27.357769 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.357748 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:28.357731027 +0000 UTC m=+4.048162825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:27.458505 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.458467 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:27.458703 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.458678 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:27.458786 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.458770 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:28.458747916 +0000 UTC m=+4.149179720 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:27.559566 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.559525 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:27.559744 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.559710 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:27.559744 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.559737 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:27.559827 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.559754 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:27.559827 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.559814 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:28.559798821 +0000 UTC m=+4.250230619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:27.788019 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.787910 2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 21:11:27 +0000 UTC" deadline="2028-01-20 10:53:17.566403003 +0000 UTC" Jul 29 21:16:27.788019 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.787955 2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12949h36m49.778451942s" Jul 29 21:16:27.847729 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:27.847694 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:27.847912 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:27.847821 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:28.029857 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.029827 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c58c126_7301_46d4_963d_f5c15a7f8712.slice/crio-cf7247501e53783b4dca2585308269c069edf2ae361284b5c87c430398d4af94 WatchSource:0}: Error finding container cf7247501e53783b4dca2585308269c069edf2ae361284b5c87c430398d4af94: Status 404 returned error can't find the container with id cf7247501e53783b4dca2585308269c069edf2ae361284b5c87c430398d4af94 Jul 29 21:16:28.031668 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.031643 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58046276_d71d_4df4_afeb_a5f37f66c360.slice/crio-26e0e2094105a13156cb3d7892c52537b53a9407d1d005ffb417b031be415265 WatchSource:0}: Error finding container 26e0e2094105a13156cb3d7892c52537b53a9407d1d005ffb417b031be415265: Status 404 returned error can't find the container with id 26e0e2094105a13156cb3d7892c52537b53a9407d1d005ffb417b031be415265 Jul 29 21:16:28.053706 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.053674 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ab5b94_4525_4d7f_b9c9_f183e6a75d26.slice/crio-d1a2025a63a05ac17c1d999aa0d97ed697219bcd55acc00e15463154c64fd98b WatchSource:0}: Error finding container d1a2025a63a05ac17c1d999aa0d97ed697219bcd55acc00e15463154c64fd98b: Status 404 returned error can't find the container with id d1a2025a63a05ac17c1d999aa0d97ed697219bcd55acc00e15463154c64fd98b Jul 29 21:16:28.054636 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.054507 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514a6c7e_c678_4089_8371_2f50ec8bd987.slice/crio-57a082f6156a53b7c92fbe219560682bc9c5914828ce8c941d497a657db333e6 WatchSource:0}: Error finding container 57a082f6156a53b7c92fbe219560682bc9c5914828ce8c941d497a657db333e6: Status 404 returned error can't find the container with id 57a082f6156a53b7c92fbe219560682bc9c5914828ce8c941d497a657db333e6 Jul 29 21:16:28.056199 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.056180 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72c190d6_a300_4510_bcab_fe14c70974cc.slice/crio-9d36e5a61d4bafcc67dca7a1c27e901731ca33e5315a811cd9ec2640ca39e995 WatchSource:0}: Error finding container 9d36e5a61d4bafcc67dca7a1c27e901731ca33e5315a811cd9ec2640ca39e995: Status 404 returned error can't find the container with id 9d36e5a61d4bafcc67dca7a1c27e901731ca33e5315a811cd9ec2640ca39e995 Jul 29 21:16:28.059570 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.059548 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e344e6_185e_4fb2_bbf9_c07db3b2e1b2.slice/crio-e74c54d2377c97f03df1b9aa2df376f7b08500936dd7752bee7793d54a6e15ee WatchSource:0}: Error finding container e74c54d2377c97f03df1b9aa2df376f7b08500936dd7752bee7793d54a6e15ee: Status 404 returned error can't find the container with id e74c54d2377c97f03df1b9aa2df376f7b08500936dd7752bee7793d54a6e15ee Jul 29 21:16:28.060879 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.060787 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946d043a_d4b9_42bc_87e5_41db6c5d7fde.slice/crio-c47e871105a3bc42d54203ec25091a0e0c7d69cec9132b83fa6a902053b0e593 WatchSource:0}: Error finding container c47e871105a3bc42d54203ec25091a0e0c7d69cec9132b83fa6a902053b0e593: Status 404 returned error can't find the container with id c47e871105a3bc42d54203ec25091a0e0c7d69cec9132b83fa6a902053b0e593 Jul 29 21:16:28.061201 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.061112 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5e37d0_5571_4a03_bb99_4cc4b08897f3.slice/crio-bd738d6af3630973bd93e0df6144138efb8bb822e372657d3cc6de8fbc9ab3ef WatchSource:0}: Error finding container bd738d6af3630973bd93e0df6144138efb8bb822e372657d3cc6de8fbc9ab3ef: Status 404 returned error can't find the container with id bd738d6af3630973bd93e0df6144138efb8bb822e372657d3cc6de8fbc9ab3ef Jul 29 21:16:28.063022 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:16:28.062990 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf31b11f5_cefa_4201_ae2b_b088d025948a.slice/crio-fda3a7a9e6527c9bf9e3538a95b558902b341e38ad9c5b736aae329deeaf7177 WatchSource:0}: Error finding container fda3a7a9e6527c9bf9e3538a95b558902b341e38ad9c5b736aae329deeaf7177: Status 404 returned error can't find the container with id fda3a7a9e6527c9bf9e3538a95b558902b341e38ad9c5b736aae329deeaf7177 Jul 29 21:16:28.369593 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.369394 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:28.369728 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.369564 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:28.369728 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.369688 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:30.369672794 +0000 UTC m=+6.060104592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:28.470961 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.470909 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:28.471135 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.471081 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:28.471180 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.471150 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:30.471134275 +0000 UTC m=+6.161566075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:28.571656 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.571556 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:28.571844 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.571729 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:28.571844 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.571749 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:28.571844 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.571765 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:28.571844 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.571829 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:30.571808205 +0000 UTC m=+6.262240015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:28.789008 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.788904 2527 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 21:11:27 +0000 UTC" deadline="2028-03-30 07:52:27.301273252 +0000 UTC" Jul 29 21:16:28.789008 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.788981 2527 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14626h35m58.51229711s" Jul 29 21:16:28.847640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.847556 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:28.850579 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.850051 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:28.850579 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.847613 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:28.850579 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:28.850521 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:28.877036 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.876499 2527 generic.go:356] "Generic (PLEG): container finished" podID="b6a7dffde4427b22d8c34d040f4ff0c1" containerID="eadbb512736eb2899a221cdd6f29e32ef25e3d5061865d4af501a7993e9556b4" exitCode=0 Jul 29 21:16:28.877036 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.876595 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" event={"ID":"b6a7dffde4427b22d8c34d040f4ff0c1","Type":"ContainerDied","Data":"eadbb512736eb2899a221cdd6f29e32ef25e3d5061865d4af501a7993e9556b4"} Jul 29 21:16:28.884197 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.884163 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" event={"ID":"9b454936a2c3aaec87ddb9b7f44eca43","Type":"ContainerStarted","Data":"5de896d581725239fa85cc3a9e2e755b30c632a715b62f135058581e14930298"} Jul 29 21:16:28.907064 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.907030 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wq45j" event={"ID":"f31b11f5-cefa-4201-ae2b-b088d025948a","Type":"ContainerStarted","Data":"fda3a7a9e6527c9bf9e3538a95b558902b341e38ad9c5b736aae329deeaf7177"} Jul 29 21:16:28.921798 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.921759 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerStarted","Data":"c47e871105a3bc42d54203ec25091a0e0c7d69cec9132b83fa6a902053b0e593"} Jul 29 21:16:28.930958 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.930924 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" event={"ID":"72c190d6-a300-4510-bcab-fe14c70974cc","Type":"ContainerStarted","Data":"9d36e5a61d4bafcc67dca7a1c27e901731ca33e5315a811cd9ec2640ca39e995"} Jul 29 21:16:28.951572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.951528 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztx7p" event={"ID":"58046276-d71d-4df4-afeb-a5f37f66c360","Type":"ContainerStarted","Data":"26e0e2094105a13156cb3d7892c52537b53a9407d1d005ffb417b031be415265"} Jul 29 21:16:28.966079 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.966034 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"cf7247501e53783b4dca2585308269c069edf2ae361284b5c87c430398d4af94"} Jul 29 21:16:28.979968 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.979888 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" event={"ID":"5d5e37d0-5571-4a03-bb99-4cc4b08897f3","Type":"ContainerStarted","Data":"bd738d6af3630973bd93e0df6144138efb8bb822e372657d3cc6de8fbc9ab3ef"} Jul 29 21:16:28.996881 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:28.996803 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fnxnn" event={"ID":"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2","Type":"ContainerStarted","Data":"e74c54d2377c97f03df1b9aa2df376f7b08500936dd7752bee7793d54a6e15ee"} Jul 29 21:16:29.005462 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:29.005431 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxx4w" event={"ID":"514a6c7e-c678-4089-8371-2f50ec8bd987","Type":"ContainerStarted","Data":"57a082f6156a53b7c92fbe219560682bc9c5914828ce8c941d497a657db333e6"} Jul 29 21:16:29.011128 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:29.011041 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8469" event={"ID":"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26","Type":"ContainerStarted","Data":"d1a2025a63a05ac17c1d999aa0d97ed697219bcd55acc00e15463154c64fd98b"} Jul 29 21:16:29.847750 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:29.847718 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:29.848272 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:29.847917 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:30.028251 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.027171 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" event={"ID":"b6a7dffde4427b22d8c34d040f4ff0c1","Type":"ContainerStarted","Data":"a47d2263cec2f9011336b6b03708f97a2a65fe23f338688473b205cbb41ab4c2"} Jul 29 21:16:30.044234 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.044166 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-233.ec2.internal" podStartSLOduration=4.044139264 podStartE2EDuration="4.044139264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:28.910146131 +0000 UTC m=+4.600577968" watchObservedRunningTime="2026-07-29 21:16:30.044139264 +0000 UTC m=+5.734571087" Jul 29 21:16:30.389256 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.388337 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:30.389256 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.388543 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:30.389256 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.388612 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:34.388590888 +0000 UTC m=+10.079022700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:30.490109 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.489154 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:30.490109 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.489475 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:30.490109 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.489553 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:34.489529202 +0000 UTC m=+10.179961006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:30.590931 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.590301 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:30.590931 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.590465 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:30.590931 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.590490 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:30.590931 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.590503 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:30.590931 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.590566 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:34.590544631 +0000 UTC m=+10.280976436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:30.848165 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.847575 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:30.848165 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:30.847613 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:30.848165 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.847716 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:30.848165 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:30.847831 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:31.847613 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:31.847577 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:31.847797 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:31.847734 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:32.850689 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:32.850040 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:32.850689 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:32.850174 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:32.850689 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:32.850541 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:32.850689 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:32.850632 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:33.847580 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:33.847542 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:33.847759 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:33.847684 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:34.424329 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:34.424291 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:34.424736 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.424469 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:34.424736 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.424538 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:42.424516832 +0000 UTC m=+18.114948629 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:34.525216 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:34.525180 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:34.525417 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.525384 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:34.525481 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.525443 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:42.52542514 +0000 UTC m=+18.215856945 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:34.626191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:34.626145 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:34.626373 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.626361 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:34.626433 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.626385 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:34.626433 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.626399 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:34.626531 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.626453 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:42.626436892 +0000 UTC m=+18.316868708 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:34.847902 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:34.847827 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:34.848075 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.847916 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:34.848075 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:34.848007 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:34.848191 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:34.848112 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:35.848069 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:35.847565 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:35.848069 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:35.847698 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:36.850196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:36.849495 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:36.850196 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:36.849631 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:36.850196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:36.850040 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:36.850196 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:36.850132 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:37.847534 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:37.847496 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:37.847694 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:37.847627 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:38.847127 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:38.847093 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:38.847520 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:38.847094 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:38.847520 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:38.847233 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:38.847520 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:38.847313 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:39.847169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:39.847128 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:39.847570 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:39.847256 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:40.847627 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:40.847595 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:40.848058 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:40.847595 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:40.848058 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:40.847742 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:40.848058 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:40.847817 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:41.847747 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:41.847716 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:41.848238 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:41.847847 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:42.499053 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:42.499017 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:42.499216 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.499171 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:42.499256 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.499239 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:16:58.499222538 +0000 UTC m=+34.189654336 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:42.599522 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:42.599486 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:42.599676 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.599602 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:42.599676 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.599654 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:16:58.599642004 +0000 UTC m=+34.290073799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:42.699914 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:42.699874 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:42.700085 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.700049 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:42.700085 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.700074 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:42.700085 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.700084 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:42.700179 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.700133 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:16:58.700120186 +0000 UTC m=+34.390551986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:42.847432 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:42.847355 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:42.847582 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.847489 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:42.847582 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:42.847528 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:42.847687 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:42.847635 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:43.846960 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:43.846913 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:43.847332 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:43.847105 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:44.847708 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:44.847672 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:44.848168 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:44.847670 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:44.848168 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:44.847820 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:44.848168 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:44.847879 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:45.847082 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:45.847051 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:45.847243 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:45.847183 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:46.847633 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:46.847598 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:46.848125 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:46.847727 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:46.848125 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:46.847774 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:46.848125 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:46.847847 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:47.847466 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:47.846996 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:47.847672 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:47.847590 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:48.071058 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.071024 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" event={"ID":"5d5e37d0-5571-4a03-bb99-4cc4b08897f3","Type":"ContainerStarted","Data":"22b985a114c79f367faa510d6e295d46de01153bc982c5ab5e0b5f12b873a739"} Jul 29 21:16:48.072407 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.072380 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-fnxnn" event={"ID":"b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2","Type":"ContainerStarted","Data":"f5e8ec6939017fde443b713fb57783ba2fed7b151c0d2cd7d0f2fbe84cf83671"} Jul 29 21:16:48.073735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.073710 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxx4w" event={"ID":"514a6c7e-c678-4089-8371-2f50ec8bd987","Type":"ContainerStarted","Data":"ce62f1ddb3171598075d84eded62f98129ef5cf1619e0702c102f3eef797708f"} Jul 29 21:16:48.074898 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.074874 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wq45j" event={"ID":"f31b11f5-cefa-4201-ae2b-b088d025948a","Type":"ContainerStarted","Data":"90691bb5dea5e5371d33aebfc528a12cc62d895bcd995d1eeba8fc3d3516c33b"} Jul 29 21:16:48.076396 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.076371 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="4bd0182640e63caad34e59dc6b4595b684a7a6ae605c0c811c674d6b4dd5c234" exitCode=0 Jul 29 21:16:48.076484 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.076445 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"4bd0182640e63caad34e59dc6b4595b684a7a6ae605c0c811c674d6b4dd5c234"} Jul 29 21:16:48.077838 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.077822 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" event={"ID":"72c190d6-a300-4510-bcab-fe14c70974cc","Type":"ContainerStarted","Data":"11bd6e15b6589dc76daa19e5ca6014003097c0f21fd406ac66980a0122281e62"} Jul 29 21:16:48.079351 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.079332 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ztx7p" event={"ID":"58046276-d71d-4df4-afeb-a5f37f66c360","Type":"ContainerStarted","Data":"650d77b7a041f67840a1584ccb4389338a7825bf41eb55e2f2153eea2d01b41b"} Jul 29 21:16:48.082067 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082051 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:16:48.082796 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082779 2527 generic.go:356] "Generic (PLEG): container finished" podID="7c58c126-7301-46d4-963d-f5c15a7f8712" containerID="014d24e555dbc3bb127556fe634a2f19d41f0f25f7a619d34ec82332826177b1" exitCode=1 Jul 29 21:16:48.082852 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082806 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"818d1c3b9bd005e39973c4c88578d544e8a2a589857facdde6b7d01641568702"} Jul 29 21:16:48.082886 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082874 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"d643390fe3310a608f4cb96b23029aaf8287a66386d4f833cdd0d70a5601acf8"} Jul 29 21:16:48.082922 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082910 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"49b66003dcbf3005a74176396df4c902784394ac42575acd28479e55d1d434d0"} Jul 29 21:16:48.082992 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082967 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"f15b9f7d4cbdd635ff0f95ecd5813f3ff71af1885115de57ec776867162844d3"} Jul 29 21:16:48.083025 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.082995 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerDied","Data":"014d24e555dbc3bb127556fe634a2f19d41f0f25f7a619d34ec82332826177b1"} Jul 29 21:16:48.083025 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.083022 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"d2a2d9fbc641377719602016f691e3f0201bd2bd62ebab687b13135fd93d805a"} Jul 29 21:16:48.092012 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.091974 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-fnxnn" podStartSLOduration=4.043759383 podStartE2EDuration="23.091961997s" totalImagesPullingTime="19.048202614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.09159013 +0000 UTC m=+23.782021948" watchObservedRunningTime="2026-07-29 21:16:48.091961997 +0000 UTC m=+23.782393812" Jul 29 21:16:48.092225 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.092207 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-233.ec2.internal" podStartSLOduration=22.092201441 podStartE2EDuration="22.092201441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:30.044701026 +0000 UTC m=+5.735132849" watchObservedRunningTime="2026-07-29 21:16:48.092201441 +0000 UTC m=+23.782633257" Jul 29 21:16:48.109263 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.109207 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qh9jq" podStartSLOduration=3.969519031 podStartE2EDuration="23.109193097s" totalImagesPullingTime="19.139674066s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.108920081 +0000 UTC m=+23.799351899" watchObservedRunningTime="2026-07-29 21:16:48.109193097 +0000 UTC m=+23.799624960" Jul 29 21:16:48.120728 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.120705 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:48.121242 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.121228 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:48.130352 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.130306 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wq45j" podStartSLOduration=12.385742433 podStartE2EDuration="23.13029268s" totalImagesPullingTime="10.744550247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.130279355 +0000 UTC m=+23.820711173" watchObservedRunningTime="2026-07-29 21:16:48.13029268 +0000 UTC m=+23.820724497" Jul 29 21:16:48.146471 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.146383 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ztx7p" podStartSLOduration=4.10659949 podStartE2EDuration="23.146369193s" totalImagesPullingTime="19.039769703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.146120687 +0000 UTC m=+23.836552503" watchObservedRunningTime="2026-07-29 21:16:48.146369193 +0000 UTC m=+23.836801009" Jul 29 21:16:48.193706 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.193650 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fxx4w" podStartSLOduration=4.179618524 podStartE2EDuration="23.193635737s" totalImagesPullingTime="19.014017213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:48.171792184 +0000 UTC m=+23.862224004" watchObservedRunningTime="2026-07-29 21:16:48.193635737 +0000 UTC m=+23.884067613" Jul 29 21:16:48.580831 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.580800 2527 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 21:16:48.804262 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.804088 2527 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T21:16:48.580826378Z","UUID":"41550ddf-6b9e-4137-a7ee-96d0becd774d","Handler":null,"Name":"","Endpoint":""} Jul 29 21:16:48.805956 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.805894 2527 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 21:16:48.805956 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.805970 2527 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 21:16:48.847252 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.847227 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:48.847424 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:48.847350 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:48.847488 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:48.847439 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:48.847556 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:48.847526 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:49.087281 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:49.087191 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" event={"ID":"5d5e37d0-5571-4a03-bb99-4cc4b08897f3","Type":"ContainerStarted","Data":"c679010ed87588f5e6387afc2e8413eb34a2292c3078031518e228f4d47d9f58"} Jul 29 21:16:49.089328 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:49.089299 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l8469" event={"ID":"f9ab5b94-4525-4d7f-b9c9-f183e6a75d26","Type":"ContainerStarted","Data":"38516df8b5220b2ca4a0e433ff1a1c9e4baecdbb4fb7119d0382ee51adcd1592"} Jul 29 21:16:49.109966 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:49.109883 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l8469" podStartSLOduration=5.073223491 podStartE2EDuration="24.109860565s" totalImagesPullingTime="19.036637074s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:49.109508407 +0000 UTC m=+24.799940227" watchObservedRunningTime="2026-07-29 21:16:49.109860565 +0000 UTC m=+24.800292385" Jul 29 21:16:49.847557 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:49.847327 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:49.847721 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:49.847611 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:50.097680 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.097597 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:16:50.098214 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.098187 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"f6f09967bfa172dc0e41c13a0deef6566fd9d7a0d929937ea1f1fc9ced52139d"} Jul 29 21:16:50.103601 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.103568 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" event={"ID":"5d5e37d0-5571-4a03-bb99-4cc4b08897f3","Type":"ContainerStarted","Data":"f57e63513f0ed7d081942f85f2df595159b431ff474cf59bc7ed72abc1337363"} Jul 29 21:16:50.103729 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.103618 2527 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 21:16:50.121140 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.121080 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hgvqw" podStartSLOduration=4.088354037 podStartE2EDuration="25.121054699s" totalImagesPullingTime="21.032700662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:50.121026266 +0000 UTC m=+25.811458087" watchObservedRunningTime="2026-07-29 21:16:50.121054699 +0000 UTC m=+25.811486515" Jul 29 21:16:50.847425 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.847387 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:50.847637 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:50.847518 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:50.847637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:50.847550 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:50.847739 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:50.847653 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:51.847316 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:51.847282 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:51.847964 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:51.847417 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:52.847415 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:52.847224 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:52.848009 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:52.847280 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:52.848009 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:52.847485 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:52.848009 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:52.847542 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:53.111651 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.111616 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="bb1fcbd929c625f7f7f6127e3c7b1feb03cbe2d96951d98fdf736507b978b85a" exitCode=0 Jul 29 21:16:53.111790 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.111696 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"bb1fcbd929c625f7f7f6127e3c7b1feb03cbe2d96951d98fdf736507b978b85a"} Jul 29 21:16:53.115323 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.115305 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:16:53.115719 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.115692 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"04d544584fe11de5f3de77a95c15d1d0df228dc8fae2232ce35dd61270e324ee"} Jul 29 21:16:53.115971 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.115953 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:53.116057 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.116005 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:53.116057 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.116043 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:53.116186 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.116160 2527 scope.go:117] "RemoveContainer" containerID="014d24e555dbc3bb127556fe634a2f19d41f0f25f7a619d34ec82332826177b1" Jul 29 21:16:53.133103 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.133081 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:53.134161 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.134140 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:16:53.847587 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:53.847557 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:53.848044 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:53.847662 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:54.121458 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.121419 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="942d400db7b8680526e9ea44353405f0a13888921acbb92af735dc5517d6d412" exitCode=0 Jul 29 21:16:54.121648 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.121497 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"942d400db7b8680526e9ea44353405f0a13888921acbb92af735dc5517d6d412"} Jul 29 21:16:54.125594 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.125572 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:16:54.126021 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.126000 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" event={"ID":"7c58c126-7301-46d4-963d-f5c15a7f8712","Type":"ContainerStarted","Data":"d0e7f3a763c6b94eda6be7a2feddcc15bc402196d617615b137f71f8ab3fd935"} Jul 29 21:16:54.180819 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.180765 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" podStartSLOduration=10.092550358 podStartE2EDuration="29.180748245s" totalImagesPullingTime="19.088197887s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:16:54.180564273 +0000 UTC m=+29.870996129" watchObservedRunningTime="2026-07-29 21:16:54.180748245 +0000 UTC m=+29.871180062" Jul 29 21:16:54.593885 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.593619 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kv7sc"] Jul 29 21:16:54.594072 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.593942 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:54.594072 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:54.594041 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.597666 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhtjs"] Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.597838 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:54.597985 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.598164 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qszv"] Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:54.598294 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:54.600134 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:54.598430 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:55.130637 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.130602 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="c670ce6af5fdeb3601ff78d76d84228b3db639ae3d0b107a50eaf838fbb1942b" exitCode=0 Jul 29 21:16:55.131172 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.130688 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"c670ce6af5fdeb3601ff78d76d84228b3db639ae3d0b107a50eaf838fbb1942b"} Jul 29 21:16:55.817373 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.817339 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:55.817564 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.817486 2527 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 21:16:55.818208 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.818189 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wq45j" Jul 29 21:16:55.847337 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:55.847308 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:55.847483 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:55.847431 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:56.847518 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:56.847427 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:56.848311 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:56.847556 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:56.848311 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:56.847611 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:56.848311 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:56.847723 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:57.847015 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:57.846985 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:57.847198 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:57.847090 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:16:58.519017 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:58.518984 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:58.519429 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.519151 2527 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:58.519429 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.519227 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs podName:2594938e-7cbe-44b6-a97b-9a8d9d08dbfc nodeName:}" failed. No retries permitted until 2026-07-29 21:17:30.519210966 +0000 UTC m=+66.209642761 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs") pod "network-metrics-daemon-9qszv" (UID: "2594938e-7cbe-44b6-a97b-9a8d9d08dbfc") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 21:16:58.619630 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:58.619586 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:58.619827 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.619704 2527 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:58.619827 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.619757 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret podName:b3743918-d652-44d4-a116-68a05a655027 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:30.619742926 +0000 UTC m=+66.310174723 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret") pod "global-pull-secret-syncer-kv7sc" (UID: "b3743918-d652-44d4-a116-68a05a655027") : object "kube-system"/"original-pull-secret" not registered Jul 29 21:16:58.720248 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:58.720203 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:58.720421 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.720394 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 21:16:58.720421 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.720415 2527 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 21:16:58.720512 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.720427 2527 projected.go:196] Error preparing data for projected volume kube-api-access-z8dt7 for pod openshift-network-diagnostics/network-check-target-zhtjs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:58.720512 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.720489 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7 podName:e645cc01-6090-4878-8d6b-c0a640ef5e7a nodeName:}" failed. No retries permitted until 2026-07-29 21:17:30.720470991 +0000 UTC m=+66.410902787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-z8dt7" (UniqueName: "kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7") pod "network-check-target-zhtjs" (UID: "e645cc01-6090-4878-8d6b-c0a640ef5e7a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 21:16:58.847259 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:58.847221 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:16:58.847443 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:58.847264 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:16:58.847443 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.847368 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9qszv" podUID="2594938e-7cbe-44b6-a97b-9a8d9d08dbfc" Jul 29 21:16:58.847559 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:58.847487 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zhtjs" podUID="e645cc01-6090-4878-8d6b-c0a640ef5e7a" Jul 29 21:16:59.847098 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:16:59.847048 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:16:59.847476 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:16:59.847179 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kv7sc" podUID="b3743918-d652-44d4-a116-68a05a655027" Jul 29 21:17:00.133778 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.133753 2527 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-233.ec2.internal" event="NodeReady" Jul 29 21:17:00.133963 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.133876 2527 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 21:17:00.148812 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.148774 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="fed106e0b04361f6b383675b998c30ec90dbcfb051e421ce0e1074761dec31ce" exitCode=0 Jul 29 21:17:00.148990 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.148838 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"fed106e0b04361f6b383675b998c30ec90dbcfb051e421ce0e1074761dec31ce"} Jul 29 21:17:00.188047 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.187973 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5zszr"] Jul 29 21:17:00.221773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.221744 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pw6fw"] Jul 29 21:17:00.221908 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.221883 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.224153 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.224127 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6msdl\"" Jul 29 21:17:00.224415 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.224400 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 21:17:00.224539 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.224525 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 21:17:00.237820 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.237724 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5zszr"] Jul 29 21:17:00.237820 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.237765 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pw6fw"] Jul 29 21:17:00.238027 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.237855 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.240394 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.240372 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 21:17:00.240570 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.240415 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-djnws\"" Jul 29 21:17:00.240570 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.240442 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 21:17:00.240570 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.240478 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 21:17:00.331595 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.331558 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.331718 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.331605 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-config-volume\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.331770 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.331727 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-tmp-dir\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.331810 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.331787 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rqn\" (UniqueName: \"kubernetes.io/projected/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-kube-api-access-w6rqn\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.432923 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-tmp-dir\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.432980 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rqn\" (UniqueName: \"kubernetes.io/projected/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-kube-api-access-w6rqn\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433226 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433037 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433226 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433056 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a838926f-fe57-4fde-9529-fbf598e51fdc-cert\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.433226 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433082 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cgr9\" (UniqueName: \"kubernetes.io/projected/a838926f-fe57-4fde-9529-fbf598e51fdc-kube-api-access-2cgr9\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.433226 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433132 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-config-volume\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433226 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:00.433191 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:00.433394 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:00.433257 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:00.933239131 +0000 UTC m=+36.623670926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:00.433394 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433321 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-tmp-dir\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.433666 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.433648 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-config-volume\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.442949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.442905 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rqn\" (UniqueName: \"kubernetes.io/projected/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-kube-api-access-w6rqn\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.534055 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.534023 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a838926f-fe57-4fde-9529-fbf598e51fdc-cert\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.534055 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.534059 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cgr9\" (UniqueName: \"kubernetes.io/projected/a838926f-fe57-4fde-9529-fbf598e51fdc-kube-api-access-2cgr9\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.536603 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.536579 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a838926f-fe57-4fde-9529-fbf598e51fdc-cert\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.546570 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.546539 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cgr9\" (UniqueName: \"kubernetes.io/projected/a838926f-fe57-4fde-9529-fbf598e51fdc-kube-api-access-2cgr9\") pod \"ingress-canary-pw6fw\" (UID: \"a838926f-fe57-4fde-9529-fbf598e51fdc\") " pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.547304 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.547284 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pw6fw" Jul 29 21:17:00.774400 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.774369 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pw6fw"] Jul 29 21:17:00.783615 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:00.783591 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda838926f_fe57_4fde_9529_fbf598e51fdc.slice/crio-e21bded726b726a8f7adcce19b17b200aacd563619942b4ebc703d403c4372f1 WatchSource:0}: Error finding container e21bded726b726a8f7adcce19b17b200aacd563619942b4ebc703d403c4372f1: Status 404 returned error can't find the container with id e21bded726b726a8f7adcce19b17b200aacd563619942b4ebc703d403c4372f1 Jul 29 21:17:00.847139 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.847109 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:17:00.847139 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.847154 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:17:00.849722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.849532 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 21:17:00.849722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.849541 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 21:17:00.849722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.849597 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 21:17:00.849722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.849545 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-929b2\"" Jul 29 21:17:00.849722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.849720 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql2h5\"" Jul 29 21:17:00.936249 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:00.936210 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:00.936388 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:00.936335 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:00.936442 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:00.936399 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:01.936383975 +0000 UTC m=+37.626815791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:01.154512 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.154484 2527 generic.go:356] "Generic (PLEG): container finished" podID="946d043a-d4b9-42bc-87e5-41db6c5d7fde" containerID="d20966b451e3d562347222c26cd54706980d73ecfaa3a6123500a715df86d7cd" exitCode=0 Jul 29 21:17:01.154693 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.154551 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerDied","Data":"d20966b451e3d562347222c26cd54706980d73ecfaa3a6123500a715df86d7cd"} Jul 29 21:17:01.159365 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.159336 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pw6fw" event={"ID":"a838926f-fe57-4fde-9529-fbf598e51fdc","Type":"ContainerStarted","Data":"e21bded726b726a8f7adcce19b17b200aacd563619942b4ebc703d403c4372f1"} Jul 29 21:17:01.847135 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.846923 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:17:01.849658 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.849635 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 21:17:01.943775 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:01.943730 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:01.943946 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:01.943883 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:01.944010 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:01.943989 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:03.943972039 +0000 UTC m=+39.634403835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:02.165273 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:02.165244 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" event={"ID":"946d043a-d4b9-42bc-87e5-41db6c5d7fde","Type":"ContainerStarted","Data":"750776a9b1bc36f6203b8c55136e3f0a954987f84270d9e3ceac4963aa62fe47"} Jul 29 21:17:02.190749 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:02.190701 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t5hk2" podStartSLOduration=7.646619662 podStartE2EDuration="37.190682809s" totalImagesPullingTime="29.544063147s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:02.189525757 +0000 UTC m=+37.879957575" watchObservedRunningTime="2026-07-29 21:17:02.190682809 +0000 UTC m=+37.881114627" Jul 29 21:17:03.956330 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:03.956289 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:03.956918 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:03.956453 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:03.956918 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:03.956533 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.956510929 +0000 UTC m=+43.646942725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:05.173825 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:05.173783 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pw6fw" event={"ID":"a838926f-fe57-4fde-9529-fbf598e51fdc","Type":"ContainerStarted","Data":"78119594b3e3679ec91ca9a9fa1fcf95e4101a4cc0655db055f18b2156f0ce62"} Jul 29 21:17:05.190646 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:05.190594 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pw6fw" podStartSLOduration=1.669355726 podStartE2EDuration="5.190578911s" totalImagesPullingTime="3.521223185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:05.189838813 +0000 UTC m=+40.880270633" watchObservedRunningTime="2026-07-29 21:17:05.190578911 +0000 UTC m=+40.881010758" Jul 29 21:17:06.881770 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.881735 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qgtcj"] Jul 29 21:17:06.903478 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.903452 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qgtcj"] Jul 29 21:17:06.903626 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.903550 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:06.905731 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.905703 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 29 21:17:06.905870 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.905729 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 29 21:17:06.906015 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.906000 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 29 21:17:06.906090 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.906048 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 29 21:17:06.906542 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.906524 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-77k5f\"" Jul 29 21:17:06.910735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.910716 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 29 21:17:06.979429 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.979393 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-trusted-ca\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:06.979606 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.979441 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mz2t\" (UniqueName: \"kubernetes.io/projected/3d0ac419-cc5c-43e3-a3ea-c690b3743268-kube-api-access-5mz2t\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:06.979606 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.979461 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d0ac419-cc5c-43e3-a3ea-c690b3743268-serving-cert\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:06.979606 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.979541 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-config\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:06.988224 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:06.988201 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv"] Jul 29 21:17:07.003548 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.003523 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv"] Jul 29 21:17:07.003666 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.003617 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.005869 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.005849 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 29 21:17:07.006045 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.005995 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-gjk84\"" Jul 29 21:17:07.006149 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.006136 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 29 21:17:07.006440 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.006423 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 29 21:17:07.079949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.079897 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-config\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.080142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.079968 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.080142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.080012 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-trusted-ca\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.080142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.080042 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mz2t\" (UniqueName: \"kubernetes.io/projected/3d0ac419-cc5c-43e3-a3ea-c690b3743268-kube-api-access-5mz2t\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.080142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.080058 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75lhc\" (UniqueName: \"kubernetes.io/projected/41b6a816-3e18-44b4-9087-ca564a5c30b1-kube-api-access-75lhc\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.080142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.080074 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d0ac419-cc5c-43e3-a3ea-c690b3743268-serving-cert\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.081096 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.081069 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-config\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.081404 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.081380 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3d0ac419-cc5c-43e3-a3ea-c690b3743268-trusted-ca\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.085053 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.085030 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3d0ac419-cc5c-43e3-a3ea-c690b3743268-serving-cert\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.089017 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.088985 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mz2t\" (UniqueName: \"kubernetes.io/projected/3d0ac419-cc5c-43e3-a3ea-c690b3743268-kube-api-access-5mz2t\") pod \"console-operator-584dcb6b54-qgtcj\" (UID: \"3d0ac419-cc5c-43e3-a3ea-c690b3743268\") " pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.093297 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.093275 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66d985499-qqtv9"] Jul 29 21:17:07.115553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.115528 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d985499-qqtv9"] Jul 29 21:17:07.115668 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.115625 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.118363 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118341 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 29 21:17:07.118494 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118341 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rrp7s\"" Jul 29 21:17:07.118494 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118414 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 29 21:17:07.118494 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118427 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 21:17:07.118674 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118661 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 21:17:07.118823 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118810 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 29 21:17:07.118868 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.118815 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 29 21:17:07.181307 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181220 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-stats-auth\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.181307 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181274 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.181307 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181301 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfqc\" (UniqueName: \"kubernetes.io/projected/d5532478-f5c9-4103-82bf-32217f059c20-kube-api-access-jgfqc\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181341 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.181381 2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181412 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75lhc\" (UniqueName: \"kubernetes.io/projected/41b6a816-3e18-44b4-9087-ca564a5c30b1-kube-api-access-75lhc\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.181437 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls podName:41b6a816-3e18-44b4-9087-ca564a5c30b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.681422523 +0000 UTC m=+43.371854318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls") pod "cluster-samples-operator-6748489656-lmkbv" (UID: "41b6a816-3e18-44b4-9087-ca564a5c30b1") : secret "samples-operator-tls" not found Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181486 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.181572 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.181517 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-default-certificate\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.190326 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.190301 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75lhc\" (UniqueName: \"kubernetes.io/projected/41b6a816-3e18-44b4-9087-ca564a5c30b1-kube-api-access-75lhc\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.213098 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.213061 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:07.282160 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.282130 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-stats-auth\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.282288 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.282210 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfqc\" (UniqueName: \"kubernetes.io/projected/d5532478-f5c9-4103-82bf-32217f059c20-kube-api-access-jgfqc\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.282288 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.282248 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.282396 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.282367 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.782354055 +0000 UTC m=+43.472785851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:07.282465 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.282393 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.282465 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.282442 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-default-certificate\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.282557 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.282518 2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 21:17:07.282606 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.282572 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:07.782555723 +0000 UTC m=+43.472987523 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : secret "router-metrics-certs-default" not found Jul 29 21:17:07.285215 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.285193 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-default-certificate\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.285895 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.285859 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-stats-auth\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.293107 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.293085 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfqc\" (UniqueName: \"kubernetes.io/projected/d5532478-f5c9-4103-82bf-32217f059c20-kube-api-access-jgfqc\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.351420 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.351382 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-qgtcj"] Jul 29 21:17:07.354237 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:07.354213 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0ac419_cc5c_43e3_a3ea_c690b3743268.slice/crio-0cab330a158905378a3e0818ab8c992b0c1de7fa297cbb40e1517cd0d2404181 WatchSource:0}: Error finding container 0cab330a158905378a3e0818ab8c992b0c1de7fa297cbb40e1517cd0d2404181: Status 404 returned error can't find the container with id 0cab330a158905378a3e0818ab8c992b0c1de7fa297cbb40e1517cd0d2404181 Jul 29 21:17:07.685068 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.685030 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:07.685258 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.685200 2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 21:17:07.685310 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.685276 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls podName:41b6a816-3e18-44b4-9087-ca564a5c30b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:08.685258797 +0000 UTC m=+44.375690593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls") pod "cluster-samples-operator-6748489656-lmkbv" (UID: "41b6a816-3e18-44b4-9087-ca564a5c30b1") : secret "samples-operator-tls" not found Jul 29 21:17:07.785563 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.785527 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.785771 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.785578 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:07.785771 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.785701 2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 21:17:07.785771 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.785729 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:08.785704544 +0000 UTC m=+44.476136356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:07.785771 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.785757 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:08.785748468 +0000 UTC m=+44.476180272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : secret "router-metrics-certs-default" not found Jul 29 21:17:07.987224 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:07.987127 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:07.987572 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.987280 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:07.987572 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:07.987342 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:15.987327673 +0000 UTC m=+51.677759468 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:08.181966 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:08.181911 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" event={"ID":"3d0ac419-cc5c-43e3-a3ea-c690b3743268","Type":"ContainerStarted","Data":"0cab330a158905378a3e0818ab8c992b0c1de7fa297cbb40e1517cd0d2404181"} Jul 29 21:17:08.692851 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:08.692813 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:08.693028 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:08.692976 2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 21:17:08.693072 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:08.693033 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls podName:41b6a816-3e18-44b4-9087-ca564a5c30b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:10.693018609 +0000 UTC m=+46.383450405 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls") pod "cluster-samples-operator-6748489656-lmkbv" (UID: "41b6a816-3e18-44b4-9087-ca564a5c30b1") : secret "samples-operator-tls" not found Jul 29 21:17:08.793591 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:08.793552 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:08.793785 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:08.793610 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:08.793785 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:08.793730 2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 21:17:08.793785 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:08.793770 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:10.79374716 +0000 UTC m=+46.484178972 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:08.793919 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:08.793801 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:10.793790985 +0000 UTC m=+46.484222785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : secret "router-metrics-certs-default" not found Jul 29 21:17:10.188608 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.188582 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/0.log" Jul 29 21:17:10.188986 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.188622 2527 generic.go:356] "Generic (PLEG): container finished" podID="3d0ac419-cc5c-43e3-a3ea-c690b3743268" containerID="a2abb37a5ad841bab1c090332136e759bef3201a48f0911079568aa3862df57c" exitCode=255 Jul 29 21:17:10.188986 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.188665 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" event={"ID":"3d0ac419-cc5c-43e3-a3ea-c690b3743268","Type":"ContainerDied","Data":"a2abb37a5ad841bab1c090332136e759bef3201a48f0911079568aa3862df57c"} Jul 29 21:17:10.188986 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.188891 2527 scope.go:117] "RemoveContainer" containerID="a2abb37a5ad841bab1c090332136e759bef3201a48f0911079568aa3862df57c" Jul 29 21:17:10.672015 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.671978 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv"] Jul 29 21:17:10.685397 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.685361 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv"] Jul 29 21:17:10.685552 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.685455 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" Jul 29 21:17:10.687785 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.687761 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-b47fk\"" Jul 29 21:17:10.710438 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.710407 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:10.710563 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.710445 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vjd\" (UniqueName: \"kubernetes.io/projected/9b82cd56-737f-43e5-8136-703371df2052-kube-api-access-f2vjd\") pod \"network-check-source-7d88cb55d8-52fvv\" (UID: \"9b82cd56-737f-43e5-8136-703371df2052\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" Jul 29 21:17:10.710599 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:10.710561 2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 21:17:10.710663 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:10.710652 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls podName:41b6a816-3e18-44b4-9087-ca564a5c30b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.710636653 +0000 UTC m=+50.401068469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls") pod "cluster-samples-operator-6748489656-lmkbv" (UID: "41b6a816-3e18-44b4-9087-ca564a5c30b1") : secret "samples-operator-tls" not found Jul 29 21:17:10.811185 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.811146 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vjd\" (UniqueName: \"kubernetes.io/projected/9b82cd56-737f-43e5-8136-703371df2052-kube-api-access-f2vjd\") pod \"network-check-source-7d88cb55d8-52fvv\" (UID: \"9b82cd56-737f-43e5-8136-703371df2052\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" Jul 29 21:17:10.811316 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.811210 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:10.811316 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.811265 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:10.811407 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:10.811382 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.811364202 +0000 UTC m=+50.501795997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:10.811460 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:10.811407 2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 21:17:10.811460 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:10.811452 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.811438317 +0000 UTC m=+50.501870114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : secret "router-metrics-certs-default" not found Jul 29 21:17:10.819091 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.819057 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vjd\" (UniqueName: \"kubernetes.io/projected/9b82cd56-737f-43e5-8136-703371df2052-kube-api-access-f2vjd\") pod \"network-check-source-7d88cb55d8-52fvv\" (UID: \"9b82cd56-737f-43e5-8136-703371df2052\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" Jul 29 21:17:10.994068 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:10.993985 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" Jul 29 21:17:11.131238 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.131207 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv"] Jul 29 21:17:11.134135 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:11.134108 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b82cd56_737f_43e5_8136_703371df2052.slice/crio-bc2018724ed68f43f61e09c678cabcdea50208cbb36bbe4775c89625dd6a27db WatchSource:0}: Error finding container bc2018724ed68f43f61e09c678cabcdea50208cbb36bbe4775c89625dd6a27db: Status 404 returned error can't find the container with id bc2018724ed68f43f61e09c678cabcdea50208cbb36bbe4775c89625dd6a27db Jul 29 21:17:11.192999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.192969 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:17:11.193398 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.193389 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/0.log" Jul 29 21:17:11.193438 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.193424 2527 generic.go:356] "Generic (PLEG): container finished" podID="3d0ac419-cc5c-43e3-a3ea-c690b3743268" containerID="b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0" exitCode=255 Jul 29 21:17:11.193471 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.193456 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" event={"ID":"3d0ac419-cc5c-43e3-a3ea-c690b3743268","Type":"ContainerDied","Data":"b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0"} Jul 29 21:17:11.193526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.193513 2527 scope.go:117] "RemoveContainer" containerID="a2abb37a5ad841bab1c090332136e759bef3201a48f0911079568aa3862df57c" Jul 29 21:17:11.193777 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.193742 2527 scope.go:117] "RemoveContainer" containerID="b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0" Jul 29 21:17:11.193972 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:11.193949 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qgtcj_openshift-console-operator(3d0ac419-cc5c-43e3-a3ea-c690b3743268)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" podUID="3d0ac419-cc5c-43e3-a3ea-c690b3743268" Jul 29 21:17:11.194736 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:11.194715 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" event={"ID":"9b82cd56-737f-43e5-8136-703371df2052","Type":"ContainerStarted","Data":"bc2018724ed68f43f61e09c678cabcdea50208cbb36bbe4775c89625dd6a27db"} Jul 29 21:17:12.199335 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.199306 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:17:12.199767 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.199689 2527 scope.go:117] "RemoveContainer" containerID="b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0" Jul 29 21:17:12.199886 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:12.199865 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qgtcj_openshift-console-operator(3d0ac419-cc5c-43e3-a3ea-c690b3743268)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" podUID="3d0ac419-cc5c-43e3-a3ea-c690b3743268" Jul 29 21:17:12.814586 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.814550 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dvbkc"] Jul 29 21:17:12.834448 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.834420 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvbkc"] Jul 29 21:17:12.834607 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.834532 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:12.836621 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.836600 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 21:17:12.837369 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.837348 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4rkn\"" Jul 29 21:17:12.837369 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.837360 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 21:17:12.837513 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.837388 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 21:17:12.837568 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.837530 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 21:17:12.928171 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.928134 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thp5p\" (UniqueName: \"kubernetes.io/projected/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-api-access-thp5p\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:12.928352 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.928217 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:12.928352 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.928253 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:12.928470 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.928390 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf33de6-8251-4745-92be-4d8e9bbdf24d-data-volume\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:12.928470 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:12.928434 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edf33de6-8251-4745-92be-4d8e9bbdf24d-crio-socket\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.029645 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.029606 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.029831 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.029652 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.029831 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.029723 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf33de6-8251-4745-92be-4d8e9bbdf24d-data-volume\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.029831 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.029779 2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 21:17:13.030019 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.029852 2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 21:17:13.030019 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.029861 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:13.529844088 +0000 UTC m=+49.220275884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : configmap "kube-rbac-proxy" not found Jul 29 21:17:13.030019 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.029957 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:13.529920607 +0000 UTC m=+49.220352605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : secret "insights-runtime-extractor-tls" not found Jul 29 21:17:13.030019 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.029978 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edf33de6-8251-4745-92be-4d8e9bbdf24d-crio-socket\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.030196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.030024 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thp5p\" (UniqueName: \"kubernetes.io/projected/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-api-access-thp5p\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.030196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.030074 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/edf33de6-8251-4745-92be-4d8e9bbdf24d-data-volume\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.030267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.030216 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/edf33de6-8251-4745-92be-4d8e9bbdf24d-crio-socket\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.038913 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.038887 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thp5p\" (UniqueName: \"kubernetes.io/projected/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-api-access-thp5p\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.534347 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.534302 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.534782 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:13.534357 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:13.534782 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.534435 2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 21:17:13.534782 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.534468 2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 21:17:13.534782 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.534522 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.534500592 +0000 UTC m=+50.224932402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : configmap "kube-rbac-proxy" not found Jul 29 21:17:13.534782 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:13.534541 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:14.534533361 +0000 UTC m=+50.224965163 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : secret "insights-runtime-extractor-tls" not found Jul 29 21:17:14.545722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.545685 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:14.545722 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.545727 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:14.546186 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.545808 2527 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 29 21:17:14.546186 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.545826 2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 21:17:14.546186 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.545874 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:16.54586179 +0000 UTC m=+52.236293587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : secret "insights-runtime-extractor-tls" not found Jul 29 21:17:14.546186 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.545894 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:16.545879591 +0000 UTC m=+52.236311387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : configmap "kube-rbac-proxy" not found Jul 29 21:17:14.746946 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.746871 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:14.747145 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.747054 2527 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 29 21:17:14.747145 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.747124 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls podName:41b6a816-3e18-44b4-9087-ca564a5c30b1 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:22.747109873 +0000 UTC m=+58.437541673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls") pod "cluster-samples-operator-6748489656-lmkbv" (UID: "41b6a816-3e18-44b4-9087-ca564a5c30b1") : secret "samples-operator-tls" not found Jul 29 21:17:14.793839 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.793814 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxx4w_514a6c7e-c678-4089-8371-2f50ec8bd987/dns-node-resolver/0.log" Jul 29 21:17:14.847407 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.847314 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:14.847407 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:14.847365 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:14.847623 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.847492 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:22.847472113 +0000 UTC m=+58.537903909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:14.847623 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.847499 2527 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 29 21:17:14.847623 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:14.847551 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:22.847534209 +0000 UTC m=+58.537966218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : secret "router-metrics-certs-default" not found Jul 29 21:17:15.209661 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:15.209624 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" event={"ID":"9b82cd56-737f-43e5-8136-703371df2052","Type":"ContainerStarted","Data":"c22a31bfbcb956692de53dd1153e312a88af7704504e553bcde8c9f0f34c6558"} Jul 29 21:17:15.225773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:15.225713 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-52fvv" podStartSLOduration=1.772677511 podStartE2EDuration="5.225692069s" totalImagesPullingTime="3.453014558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:15.224529492 +0000 UTC m=+50.914961310" watchObservedRunningTime="2026-07-29 21:17:15.225692069 +0000 UTC m=+50.916123888" Jul 29 21:17:15.794324 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:15.794295 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztx7p_58046276-d71d-4df4-afeb-a5f37f66c360/node-ca/0.log" Jul 29 21:17:16.055358 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:16.055267 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:16.055499 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:16.055413 2527 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 29 21:17:16.055499 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:16.055479 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls podName:e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:32.055460485 +0000 UTC m=+67.745892281 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls") pod "dns-default-5zszr" (UID: "e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4") : secret "dns-default-metrics-tls" not found Jul 29 21:17:16.559247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:16.559210 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:16.559247 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:16.559250 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:16.559461 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:16.559420 2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 21:17:16.559496 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:16.559478 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:20.559462926 +0000 UTC m=+56.249894742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : secret "insights-runtime-extractor-tls" not found Jul 29 21:17:16.559760 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:16.559739 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/edf33de6-8251-4745-92be-4d8e9bbdf24d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:16.595413 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:16.595381 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pw6fw_a838926f-fe57-4fde-9529-fbf598e51fdc/serve-healthcheck-canary/0.log" Jul 29 21:17:17.213180 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:17.213140 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:17.213742 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:17.213210 2527 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:17.213742 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:17.213560 2527 scope.go:117] "RemoveContainer" containerID="b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0" Jul 29 21:17:17.213742 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:17.213707 2527 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-qgtcj_openshift-console-operator(3d0ac419-cc5c-43e3-a3ea-c690b3743268)\"" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" podUID="3d0ac419-cc5c-43e3-a3ea-c690b3743268" Jul 29 21:17:20.594308 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:20.594269 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:20.594810 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:20.594413 2527 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 29 21:17:20.594810 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:20.594478 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls podName:edf33de6-8251-4745-92be-4d8e9bbdf24d nodeName:}" failed. No retries permitted until 2026-07-29 21:17:28.594462863 +0000 UTC m=+64.284894663 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dvbkc" (UID: "edf33de6-8251-4745-92be-4d8e9bbdf24d") : secret "insights-runtime-extractor-tls" not found Jul 29 21:17:22.810180 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.810119 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:22.813165 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.813133 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/41b6a816-3e18-44b4-9087-ca564a5c30b1-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-lmkbv\" (UID: \"41b6a816-3e18-44b4-9087-ca564a5c30b1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:22.910713 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.910671 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:22.910984 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.910745 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:22.910984 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:17:22.910888 2527 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle podName:d5532478-f5c9-4103-82bf-32217f059c20 nodeName:}" failed. No retries permitted until 2026-07-29 21:17:38.910868164 +0000 UTC m=+74.601299960 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle") pod "router-default-66d985499-qqtv9" (UID: "d5532478-f5c9-4103-82bf-32217f059c20") : configmap references non-existent config key: service-ca.crt Jul 29 21:17:22.912396 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.912370 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" Jul 29 21:17:22.913385 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:22.913362 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5532478-f5c9-4103-82bf-32217f059c20-metrics-certs\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:23.038513 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:23.038482 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv"] Jul 29 21:17:23.231157 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:23.231119 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" event={"ID":"41b6a816-3e18-44b4-9087-ca564a5c30b1","Type":"ContainerStarted","Data":"2749be3c94cddb6a8b2abf1c005d1022d4bcf6e9039b12e251649655104979f5"} Jul 29 21:17:25.142261 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:25.142230 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-djvst" Jul 29 21:17:26.242526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:26.242493 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" event={"ID":"41b6a816-3e18-44b4-9087-ca564a5c30b1","Type":"ContainerStarted","Data":"d1c4ab93617aa39b5e8bf07d9e19c5f06577e50f7f0eaf4be5bd7255263fee7a"} Jul 29 21:17:26.242916 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:26.242575 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" event={"ID":"41b6a816-3e18-44b4-9087-ca564a5c30b1","Type":"ContainerStarted","Data":"417baad85926a14709ecdd89922531b1494858c52285cea0172400d720e60d37"} Jul 29 21:17:26.258817 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:26.258763 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-lmkbv" podStartSLOduration=18.138423697 podStartE2EDuration="20.258746823s" totalImagesPullingTime="2.120323126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:26.258336032 +0000 UTC m=+61.948767849" watchObservedRunningTime="2026-07-29 21:17:26.258746823 +0000 UTC m=+61.949178637" Jul 29 21:17:28.662321 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:28.662267 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:28.664729 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:28.664710 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/edf33de6-8251-4745-92be-4d8e9bbdf24d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dvbkc\" (UID: \"edf33de6-8251-4745-92be-4d8e9bbdf24d\") " pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:28.747728 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:28.747699 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4rkn\"" Jul 29 21:17:28.756759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:28.756726 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dvbkc" Jul 29 21:17:28.885283 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:28.885252 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dvbkc"] Jul 29 21:17:28.888472 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:28.888445 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf33de6_8251_4745_92be_4d8e9bbdf24d.slice/crio-8998ab78a1bc4a8f56ecdf9926491fedb2e5f0b727b2c22987c1799fb190ce3f WatchSource:0}: Error finding container 8998ab78a1bc4a8f56ecdf9926491fedb2e5f0b727b2c22987c1799fb190ce3f: Status 404 returned error can't find the container with id 8998ab78a1bc4a8f56ecdf9926491fedb2e5f0b727b2c22987c1799fb190ce3f Jul 29 21:17:29.253158 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:29.253120 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvbkc" event={"ID":"edf33de6-8251-4745-92be-4d8e9bbdf24d","Type":"ContainerStarted","Data":"8998ab78a1bc4a8f56ecdf9926491fedb2e5f0b727b2c22987c1799fb190ce3f"} Jul 29 21:17:30.580724 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.580684 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:17:30.583360 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.583332 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 21:17:30.594455 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.594426 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2594938e-7cbe-44b6-a97b-9a8d9d08dbfc-metrics-certs\") pod \"network-metrics-daemon-9qszv\" (UID: \"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc\") " pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:17:30.682153 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.682112 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:17:30.684565 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.684541 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 21:17:30.697657 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.697622 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b3743918-d652-44d4-a116-68a05a655027-original-pull-secret\") pod \"global-pull-secret-syncer-kv7sc\" (UID: \"b3743918-d652-44d4-a116-68a05a655027\") " pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:17:30.783436 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.783401 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:17:30.786662 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.786634 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8dt7\" (UniqueName: \"kubernetes.io/projected/e645cc01-6090-4878-8d6b-c0a640ef5e7a-kube-api-access-z8dt7\") pod \"network-check-target-zhtjs\" (UID: \"e645cc01-6090-4878-8d6b-c0a640ef5e7a\") " pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:17:30.860315 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.860282 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-ql2h5\"" Jul 29 21:17:30.865443 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.865420 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-929b2\"" Jul 29 21:17:30.868327 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.868300 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9qszv" Jul 29 21:17:30.873528 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.873502 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:17:30.956148 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:30.955845 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kv7sc" Jul 29 21:17:31.048170 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.047637 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9qszv"] Jul 29 21:17:31.054638 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:31.054610 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2594938e_7cbe_44b6_a97b_9a8d9d08dbfc.slice/crio-3c6b0ec3c1965136f9f3c79dc88949ddb3b5464ed52e7041b4a9f889136bc5f5 WatchSource:0}: Error finding container 3c6b0ec3c1965136f9f3c79dc88949ddb3b5464ed52e7041b4a9f889136bc5f5: Status 404 returned error can't find the container with id 3c6b0ec3c1965136f9f3c79dc88949ddb3b5464ed52e7041b4a9f889136bc5f5 Jul 29 21:17:31.065819 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.065795 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zhtjs"] Jul 29 21:17:31.069276 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:31.069240 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode645cc01_6090_4878_8d6b_c0a640ef5e7a.slice/crio-1386c292a1a2b0a1b9bf8500a17d736b4d402e9faf735c5c1f7590d6324ef266 WatchSource:0}: Error finding container 1386c292a1a2b0a1b9bf8500a17d736b4d402e9faf735c5c1f7590d6324ef266: Status 404 returned error can't find the container with id 1386c292a1a2b0a1b9bf8500a17d736b4d402e9faf735c5c1f7590d6324ef266 Jul 29 21:17:31.128821 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.128793 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kv7sc"] Jul 29 21:17:31.134328 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:31.134294 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3743918_d652_44d4_a116_68a05a655027.slice/crio-a1c93a8bebf2fd451a2844d7a63c5780645a0ad2a0a8753eaa019aedc39dc21f WatchSource:0}: Error finding container a1c93a8bebf2fd451a2844d7a63c5780645a0ad2a0a8753eaa019aedc39dc21f: Status 404 returned error can't find the container with id a1c93a8bebf2fd451a2844d7a63c5780645a0ad2a0a8753eaa019aedc39dc21f Jul 29 21:17:31.261994 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.261957 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhtjs" event={"ID":"e645cc01-6090-4878-8d6b-c0a640ef5e7a","Type":"ContainerStarted","Data":"8d67723a8ecf8080f436d9136a8da58248a0e4b7a778ceeb9a15d8e271977688"} Jul 29 21:17:31.262240 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.262016 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zhtjs" event={"ID":"e645cc01-6090-4878-8d6b-c0a640ef5e7a","Type":"ContainerStarted","Data":"1386c292a1a2b0a1b9bf8500a17d736b4d402e9faf735c5c1f7590d6324ef266"} Jul 29 21:17:31.262240 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.262065 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:17:31.263154 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.263129 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kv7sc" event={"ID":"b3743918-d652-44d4-a116-68a05a655027","Type":"ContainerStarted","Data":"a1c93a8bebf2fd451a2844d7a63c5780645a0ad2a0a8753eaa019aedc39dc21f"} Jul 29 21:17:31.264378 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.264356 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qszv" event={"ID":"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc","Type":"ContainerStarted","Data":"3c6b0ec3c1965136f9f3c79dc88949ddb3b5464ed52e7041b4a9f889136bc5f5"} Jul 29 21:17:31.294194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:31.293834 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zhtjs" podStartSLOduration=66.293811723 podStartE2EDuration="1m6.293811723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:31.292217514 +0000 UTC m=+66.982649332" watchObservedRunningTime="2026-07-29 21:17:31.293811723 +0000 UTC m=+66.984243543" Jul 29 21:17:32.095523 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:32.095482 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:32.098369 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:32.098338 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4-metrics-tls\") pod \"dns-default-5zszr\" (UID: \"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4\") " pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:32.344279 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:32.344249 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6msdl\"" Jul 29 21:17:32.352256 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:32.352232 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:32.848210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:32.848136 2527 scope.go:117] "RemoveContainer" containerID="b07db3687e1e5d2bebf2f19b1c2811a24b3615080560a926178e51ada44532d0" Jul 29 21:17:34.400350 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:34.400317 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5zszr"] Jul 29 21:17:34.570572 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:34.570492 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4165d5f_2b62_4d2b_9b17_9fbc73d0dad4.slice/crio-146ab906eae26730c106e9d503bec9e5a0bca0dcb9ecb82a1034cd451838c6b7 WatchSource:0}: Error finding container 146ab906eae26730c106e9d503bec9e5a0bca0dcb9ecb82a1034cd451838c6b7: Status 404 returned error can't find the container with id 146ab906eae26730c106e9d503bec9e5a0bca0dcb9ecb82a1034cd451838c6b7 Jul 29 21:17:35.287309 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.286479 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qszv" event={"ID":"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc","Type":"ContainerStarted","Data":"9dfcc1efa0ae6382697058a9289b1325bdbddbff515810597955f69c993687be"} Jul 29 21:17:35.287309 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.286549 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9qszv" event={"ID":"2594938e-7cbe-44b6-a97b-9a8d9d08dbfc","Type":"ContainerStarted","Data":"c14fb2a69108d4897761198d18a454679720f3a6143ea22991113563b85b44c0"} Jul 29 21:17:35.289650 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.289566 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zszr" event={"ID":"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4","Type":"ContainerStarted","Data":"146ab906eae26730c106e9d503bec9e5a0bca0dcb9ecb82a1034cd451838c6b7"} Jul 29 21:17:35.292391 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.292138 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:17:35.292391 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.292235 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" event={"ID":"3d0ac419-cc5c-43e3-a3ea-c690b3743268","Type":"ContainerStarted","Data":"0c934d412943e0209ca664216025da8afa5cf35ee267c65aff374b6bae428864"} Jul 29 21:17:35.294092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.292694 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:35.296328 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.296304 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvbkc" event={"ID":"edf33de6-8251-4745-92be-4d8e9bbdf24d","Type":"ContainerStarted","Data":"acf676cdaf72cef8fe19148c9fb48e070e849325c5ccdb2236d282f88d68948f"} Jul 29 21:17:35.300212 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.300163 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" Jul 29 21:17:35.311365 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:35.311008 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9qszv" podStartSLOduration=66.746997402 podStartE2EDuration="1m10.310987213s" totalImagesPullingTime="3.563989811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:35.30899522 +0000 UTC m=+70.999427041" watchObservedRunningTime="2026-07-29 21:17:35.310987213 +0000 UTC m=+71.001419032" Jul 29 21:17:38.582553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.582486 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-qgtcj" podStartSLOduration=29.95251409 podStartE2EDuration="32.582462863s" totalImagesPullingTime="2.629948773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:35.33366946 +0000 UTC m=+71.024101279" watchObservedRunningTime="2026-07-29 21:17:38.582462863 +0000 UTC m=+74.272894681" Jul 29 21:17:38.582990 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.582645 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk"] Jul 29 21:17:38.594239 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.594213 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.597446 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.597418 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 29 21:17:38.597730 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.597714 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 29 21:17:38.598027 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.598006 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 29 21:17:38.598445 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.598423 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 29 21:17:38.598720 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.598702 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 29 21:17:38.600210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.600191 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 29 21:17:38.601438 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.601421 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 29 21:17:38.610974 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.610945 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk"] Jul 29 21:17:38.755906 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.755863 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.756085 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.755999 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55d79e9-cf8a-4653-bcf6-c37d3f451100-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.756162 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.756083 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.756224 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.756196 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8gm6\" (UniqueName: \"kubernetes.io/projected/e55d79e9-cf8a-4653-bcf6-c37d3f451100-kube-api-access-t8gm6\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.756277 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.756247 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.756378 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.756335 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857696 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857662 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857893 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857705 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857893 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857724 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55d79e9-cf8a-4653-bcf6-c37d3f451100-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857893 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857757 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857893 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857805 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8gm6\" (UniqueName: \"kubernetes.io/projected/e55d79e9-cf8a-4653-bcf6-c37d3f451100-kube-api-access-t8gm6\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.857893 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.857838 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.858510 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.858487 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e55d79e9-cf8a-4653-bcf6-c37d3f451100-additional-service-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.860766 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.860691 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-ca\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.861069 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.861028 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.861069 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.861033 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.861223 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.861075 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e55d79e9-cf8a-4653-bcf6-c37d3f451100-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.894219 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.894155 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8gm6\" (UniqueName: \"kubernetes.io/projected/e55d79e9-cf8a-4653-bcf6-c37d3f451100-kube-api-access-t8gm6\") pod \"cluster-proxy-proxy-agent-75bb8d7b75-x2zxk\" (UID: \"e55d79e9-cf8a-4653-bcf6-c37d3f451100\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.915129 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.915086 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" Jul 29 21:17:38.959156 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.959123 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:38.959814 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:38.959793 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5532478-f5c9-4103-82bf-32217f059c20-service-ca-bundle\") pod \"router-default-66d985499-qqtv9\" (UID: \"d5532478-f5c9-4103-82bf-32217f059c20\") " pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:39.226482 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:39.226395 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-rrp7s\"" Jul 29 21:17:39.234646 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:39.234613 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:40.219143 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.219045 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk"] Jul 29 21:17:40.224132 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:40.224075 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode55d79e9_cf8a_4653_bcf6_c37d3f451100.slice/crio-af66ad2b71d6f7fc4be1a492c90af94ced6a3d8e5771e7bc5991b70785b527d6 WatchSource:0}: Error finding container af66ad2b71d6f7fc4be1a492c90af94ced6a3d8e5771e7bc5991b70785b527d6: Status 404 returned error can't find the container with id af66ad2b71d6f7fc4be1a492c90af94ced6a3d8e5771e7bc5991b70785b527d6 Jul 29 21:17:40.245339 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.245306 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66d985499-qqtv9"] Jul 29 21:17:40.249961 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:40.249916 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5532478_f5c9_4103_82bf_32217f059c20.slice/crio-76db0b09aef40698e6f81aecb66d4efa4a29636ad2f73e8cff75ce8e8b831489 WatchSource:0}: Error finding container 76db0b09aef40698e6f81aecb66d4efa4a29636ad2f73e8cff75ce8e8b831489: Status 404 returned error can't find the container with id 76db0b09aef40698e6f81aecb66d4efa4a29636ad2f73e8cff75ce8e8b831489 Jul 29 21:17:40.318611 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.318570 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kv7sc" event={"ID":"b3743918-d652-44d4-a116-68a05a655027","Type":"ContainerStarted","Data":"0a0637d711ed9c97dd313b536f723186b560b233b6ec33b079b6881600d53442"} Jul 29 21:17:40.321773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.321667 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zszr" event={"ID":"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4","Type":"ContainerStarted","Data":"a26294c076c2709c85636f72b8e6bf812d100937a42ac6ac457fe4be8811ff4c"} Jul 29 21:17:40.321773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.321735 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5zszr" event={"ID":"e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4","Type":"ContainerStarted","Data":"3e4721fa286e39ef0ab3575837b215dcfaafabdaddeca31862e9305af9a507bb"} Jul 29 21:17:40.321997 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.321880 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:40.324814 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.324792 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" event={"ID":"e55d79e9-cf8a-4653-bcf6-c37d3f451100","Type":"ContainerStarted","Data":"af66ad2b71d6f7fc4be1a492c90af94ced6a3d8e5771e7bc5991b70785b527d6"} Jul 29 21:17:40.327426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.326649 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d985499-qqtv9" event={"ID":"d5532478-f5c9-4103-82bf-32217f059c20","Type":"ContainerStarted","Data":"0be8c5f9dc21904cdb15a2764391cfade12698d74b73ea58ebea6dde05706793"} Jul 29 21:17:40.327426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.326694 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66d985499-qqtv9" event={"ID":"d5532478-f5c9-4103-82bf-32217f059c20","Type":"ContainerStarted","Data":"76db0b09aef40698e6f81aecb66d4efa4a29636ad2f73e8cff75ce8e8b831489"} Jul 29 21:17:40.333612 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.333578 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvbkc" event={"ID":"edf33de6-8251-4745-92be-4d8e9bbdf24d","Type":"ContainerStarted","Data":"68ca8bc4660cea298aca530dac14884d4b8ef217e24294e6cbce8f75fcdb3b7f"} Jul 29 21:17:40.342951 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.341660 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kv7sc" podStartSLOduration=66.457057841 podStartE2EDuration="1m15.341638456s" totalImagesPullingTime="8.884580615s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:40.340980683 +0000 UTC m=+76.031412492" watchObservedRunningTime="2026-07-29 21:17:40.341638456 +0000 UTC m=+76.032070275" Jul 29 21:17:40.376195 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.376129 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5zszr" podStartSLOduration=34.927244559 podStartE2EDuration="40.37607884s" totalImagesPullingTime="5.448834281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:40.374806789 +0000 UTC m=+76.065238604" watchObservedRunningTime="2026-07-29 21:17:40.37607884 +0000 UTC m=+76.066510658" Jul 29 21:17:40.401022 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:40.400964 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66d985499-qqtv9" podStartSLOduration=33.400944418 podStartE2EDuration="33.400944418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:40.400313769 +0000 UTC m=+76.090745594" watchObservedRunningTime="2026-07-29 21:17:40.400944418 +0000 UTC m=+76.091376230" Jul 29 21:17:41.235010 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:41.234971 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:41.238038 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:41.237824 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:41.340679 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:41.340648 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:41.342045 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:41.341976 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66d985499-qqtv9" Jul 29 21:17:46.361825 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.361777 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" event={"ID":"e55d79e9-cf8a-4653-bcf6-c37d3f451100","Type":"ContainerStarted","Data":"cc157b5e0b5fd61a8fbf34c8d67634789abda05a1c9d0eb265587fb8c1f0a602"} Jul 29 21:17:46.361825 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.361827 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" event={"ID":"e55d79e9-cf8a-4653-bcf6-c37d3f451100","Type":"ContainerStarted","Data":"9ce03f33cfca92657dd98a5a821e87042a4c1cedbd55cbfc5bf8d5b10b97ed96"} Jul 29 21:17:46.362326 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.361840 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" event={"ID":"e55d79e9-cf8a-4653-bcf6-c37d3f451100","Type":"ContainerStarted","Data":"a6bce28055969f4d7f97a7a6be24d7fa03f5f34ea0f57c9749c1983167ffd041"} Jul 29 21:17:46.363721 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.363686 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dvbkc" event={"ID":"edf33de6-8251-4745-92be-4d8e9bbdf24d","Type":"ContainerStarted","Data":"29c0e9ecfbb32e8779fab1c581e159c1341e1aa1e1467e973e6ce3a2a046901d"} Jul 29 21:17:46.380962 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.380896 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-75bb8d7b75-x2zxk" podStartSLOduration=2.946983392 podStartE2EDuration="8.380881101s" totalImagesPullingTime="5.433897709s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:46.380070734 +0000 UTC m=+82.070502551" watchObservedRunningTime="2026-07-29 21:17:46.380881101 +0000 UTC m=+82.071312946" Jul 29 21:17:46.397025 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:46.396979 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dvbkc" podStartSLOduration=17.840706259 podStartE2EDuration="34.396963156s" totalImagesPullingTime="16.556256897s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 21:17:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:17:46.39621018 +0000 UTC m=+82.086642011" watchObservedRunningTime="2026-07-29 21:17:46.396963156 +0000 UTC m=+82.087394974" Jul 29 21:17:50.342992 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:50.342964 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5zszr" Jul 29 21:17:56.905655 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.905624 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9vn7r"] Jul 29 21:17:56.910314 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.910295 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:56.920938 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.920907 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 21:17:56.921210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.921194 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 21:17:56.921464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.921451 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 21:17:56.923670 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.923657 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 21:17:56.923917 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.923903 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 21:17:56.925605 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.925587 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 21:17:56.925889 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:56.925871 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-wblq4\"" Jul 29 21:17:57.001948 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.001879 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-tls\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.001948 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.001951 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-textfile\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.001981 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/47e0e612-daa8-49e2-a556-b7418658969f-kube-api-access-g28nv\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002002 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-sys\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002025 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002074 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-wtmp\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002114 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002142 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-root\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.002169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.002156 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-metrics-client-ca\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.102957 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.102879 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-sys\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.102957 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.102955 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.102979 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-wtmp\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.102993 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-sys\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103008 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103054 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-root\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103078 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-metrics-client-ca\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103103 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-tls\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103123 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-textfile\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103143 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/47e0e612-daa8-49e2-a556-b7418658969f-kube-api-access-g28nv\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103177 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-wtmp\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103220 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103175 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/47e0e612-daa8-49e2-a556-b7418658969f-root\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103641 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103562 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-textfile\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103641 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103612 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-accelerators-collector-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.103717 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.103697 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/47e0e612-daa8-49e2-a556-b7418658969f-metrics-client-ca\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.105593 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.105572 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.105789 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.105770 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/47e0e612-daa8-49e2-a556-b7418658969f-node-exporter-tls\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.122884 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.122857 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28nv\" (UniqueName: \"kubernetes.io/projected/47e0e612-daa8-49e2-a556-b7418658969f-kube-api-access-g28nv\") pod \"node-exporter-9vn7r\" (UID: \"47e0e612-daa8-49e2-a556-b7418658969f\") " pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.218426 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.218339 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9vn7r" Jul 29 21:17:57.230500 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:17:57.230460 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e0e612_daa8_49e2_a556_b7418658969f.slice/crio-df3bbfd9b1018bf931377745db94b3d62ba907d42c38bba78d3c2024c73bab95 WatchSource:0}: Error finding container df3bbfd9b1018bf931377745db94b3d62ba907d42c38bba78d3c2024c73bab95: Status 404 returned error can't find the container with id df3bbfd9b1018bf931377745db94b3d62ba907d42c38bba78d3c2024c73bab95 Jul 29 21:17:57.402810 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:57.402778 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vn7r" event={"ID":"47e0e612-daa8-49e2-a556-b7418658969f","Type":"ContainerStarted","Data":"df3bbfd9b1018bf931377745db94b3d62ba907d42c38bba78d3c2024c73bab95"} Jul 29 21:17:59.410645 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:59.410610 2527 generic.go:356] "Generic (PLEG): container finished" podID="47e0e612-daa8-49e2-a556-b7418658969f" containerID="e0ff1872ee724c6817100e3bdb9e4cfdf23f93af2f3e541b76f23590fab6ebc3" exitCode=0 Jul 29 21:17:59.411039 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:17:59.410701 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vn7r" event={"ID":"47e0e612-daa8-49e2-a556-b7418658969f","Type":"ContainerDied","Data":"e0ff1872ee724c6817100e3bdb9e4cfdf23f93af2f3e541b76f23590fab6ebc3"} Jul 29 21:18:00.416060 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:00.416026 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vn7r" event={"ID":"47e0e612-daa8-49e2-a556-b7418658969f","Type":"ContainerStarted","Data":"6daf96452adf6dd0b02a5ddfe03e28532482e6dbf2ef517ca1e4ce1ff6aa99b0"} Jul 29 21:18:00.416574 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:00.416079 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9vn7r" event={"ID":"47e0e612-daa8-49e2-a556-b7418658969f","Type":"ContainerStarted","Data":"c71751a4ecadb138167b31bb00a90ca02a33762af9033e39ddebf73a1f2a4724"} Jul 29 21:18:00.436217 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:00.436165 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9vn7r" podStartSLOduration=3.003030347 podStartE2EDuration="4.436144083s" totalImagesPullingTime="1.363353408s" totalInitContainerRuntime="69.760328ms" isStatefulPod=true podCreationTimestamp="2026-07-29 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:00.434618032 +0000 UTC m=+96.125049849" watchObservedRunningTime="2026-07-29 21:18:00.436144083 +0000 UTC m=+96.126575906" Jul 29 21:18:01.436658 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.436629 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-cc865447c-vcrtl"] Jul 29 21:18:01.441085 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.441068 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.444194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.444173 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 29 21:18:01.444194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.444187 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 29 21:18:01.444392 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.444337 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8wtjs\"" Jul 29 21:18:01.445094 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.445048 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 21:18:01.445202 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.445096 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9rfa2li1jk0h8\"" Jul 29 21:18:01.445202 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.445134 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 29 21:18:01.452841 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.452814 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cc865447c-vcrtl"] Jul 29 21:18:01.536221 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536165 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c65fdaa6-0445-425f-84f9-c7dde76130af-audit-log\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536235 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-tls\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536263 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536316 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-client-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536368 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zctlv\" (UniqueName: \"kubernetes.io/projected/c65fdaa6-0445-425f-84f9-c7dde76130af-kube-api-access-zctlv\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536551 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536390 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-metrics-server-audit-profiles\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.536551 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.536440 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-client-certs\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637748 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637712 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c65fdaa6-0445-425f-84f9-c7dde76130af-audit-log\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637758 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-tls\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637785 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637814 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-client-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637850 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zctlv\" (UniqueName: \"kubernetes.io/projected/c65fdaa6-0445-425f-84f9-c7dde76130af-kube-api-access-zctlv\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637876 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-metrics-server-audit-profiles\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.637977 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.637903 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-client-certs\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.638489 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.638463 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c65fdaa6-0445-425f-84f9-c7dde76130af-audit-log\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.638817 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.638665 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.639269 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.639247 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c65fdaa6-0445-425f-84f9-c7dde76130af-metrics-server-audit-profiles\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.640447 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.640420 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-client-ca-bundle\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.640546 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.640509 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-tls\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.640610 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.640575 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c65fdaa6-0445-425f-84f9-c7dde76130af-secret-metrics-server-client-certs\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.644510 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.644491 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:18:01.650045 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.650028 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.650154 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.650133 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zctlv\" (UniqueName: \"kubernetes.io/projected/c65fdaa6-0445-425f-84f9-c7dde76130af-kube-api-access-zctlv\") pod \"metrics-server-cc865447c-vcrtl\" (UID: \"c65fdaa6-0445-425f-84f9-c7dde76130af\") " pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.652100 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.652081 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 21:18:01.652785 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.652768 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 21:18:01.652869 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.652815 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 21:18:01.652869 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.652851 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 21:18:01.653089 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.653067 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 21:18:01.653198 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.653096 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qrndx\"" Jul 29 21:18:01.653198 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.653102 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 21:18:01.653331 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.653278 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 21:18:01.663142 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.663122 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 21:18:01.663949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.663910 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:18:01.739137 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739051 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739137 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739090 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739137 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739114 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739371 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739188 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzvl4\" (UniqueName: \"kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739371 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739229 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739371 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739259 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.739371 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.739285 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.750104 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.750074 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:01.840485 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840453 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840623 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840504 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzvl4\" (UniqueName: \"kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840623 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840545 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840623 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840592 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840851 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840625 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840851 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840655 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.840851 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.840684 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.841626 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.841599 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.841739 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.841724 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.841796 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.841760 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.841990 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.841952 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.843915 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.843878 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.844894 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.844872 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.855592 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.855560 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzvl4\" (UniqueName: \"kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4\") pod \"console-8475b99f8b-jt79f\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:01.900400 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.900369 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-cc865447c-vcrtl"] Jul 29 21:18:01.903794 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:18:01.903761 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65fdaa6_0445_425f_84f9_c7dde76130af.slice/crio-38ed67b698d6ba37e50f69573949b46c02ce23dcce31ced73733611fb2d966d0 WatchSource:0}: Error finding container 38ed67b698d6ba37e50f69573949b46c02ce23dcce31ced73733611fb2d966d0: Status 404 returned error can't find the container with id 38ed67b698d6ba37e50f69573949b46c02ce23dcce31ced73733611fb2d966d0 Jul 29 21:18:01.966066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:01.966025 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:02.137536 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:02.137512 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:18:02.139804 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:18:02.139776 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7516792_a4a5_4bcf_a975_54e13d273166.slice/crio-f99084437dc0743447264ac8ab3e8408c120b7edff6f1b8103657bc7d7943f67 WatchSource:0}: Error finding container f99084437dc0743447264ac8ab3e8408c120b7edff6f1b8103657bc7d7943f67: Status 404 returned error can't find the container with id f99084437dc0743447264ac8ab3e8408c120b7edff6f1b8103657bc7d7943f67 Jul 29 21:18:02.270370 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:02.270288 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zhtjs" Jul 29 21:18:02.423862 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:02.423830 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" event={"ID":"c65fdaa6-0445-425f-84f9-c7dde76130af","Type":"ContainerStarted","Data":"38ed67b698d6ba37e50f69573949b46c02ce23dcce31ced73733611fb2d966d0"} Jul 29 21:18:02.424949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:02.424911 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475b99f8b-jt79f" event={"ID":"a7516792-a4a5-4bcf-a975-54e13d273166","Type":"ContainerStarted","Data":"f99084437dc0743447264ac8ab3e8408c120b7edff6f1b8103657bc7d7943f67"} Jul 29 21:18:05.439264 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:05.439219 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" event={"ID":"c65fdaa6-0445-425f-84f9-c7dde76130af","Type":"ContainerStarted","Data":"33936655d9f5acf0969e4a233a526d8c6f9e0fbf09f0af38913b48e19c8254d0"} Jul 29 21:18:05.440959 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:05.440909 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475b99f8b-jt79f" event={"ID":"a7516792-a4a5-4bcf-a975-54e13d273166","Type":"ContainerStarted","Data":"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db"} Jul 29 21:18:05.457403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:05.457356 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" podStartSLOduration=1.3751513499999999 podStartE2EDuration="4.457338308s" totalImagesPullingTime="3.082186958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:05.456606028 +0000 UTC m=+101.147037847" watchObservedRunningTime="2026-07-29 21:18:05.457338308 +0000 UTC m=+101.147770128" Jul 29 21:18:05.475863 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:05.475802 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8475b99f8b-jt79f" podStartSLOduration=1.5763834220000001 podStartE2EDuration="4.475780779s" totalImagesPullingTime="2.899397357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:18:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:18:05.475703355 +0000 UTC m=+101.166135175" watchObservedRunningTime="2026-07-29 21:18:05.475780779 +0000 UTC m=+101.166212597" Jul 29 21:18:11.966605 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:11.966569 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:11.967063 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:11.966640 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:11.971342 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:11.971319 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:12.470828 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:12.470797 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:18:21.750480 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:21.750426 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:21.751030 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:21.750514 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:41.755896 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:41.755867 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:18:41.759796 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:18:41.759772 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-cc865447c-vcrtl" Jul 29 21:19:26.025756 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.025674 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:19:26.029268 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.029236 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.045322 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.045290 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:19:26.083726 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083687 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkkhk\" (UniqueName: \"kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.083900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083734 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.083900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083821 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.083900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083866 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.083900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083886 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.084106 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083910 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.084106 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.083953 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.184952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.184884 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.184952 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.184951 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.184979 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkkhk\" (UniqueName: \"kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185006 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185044 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185059 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185077 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185731 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185699 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.185954 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.185908 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.186059 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.186043 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.186103 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.186056 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.187689 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.187658 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.187779 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.187764 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.193530 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.193507 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkkhk\" (UniqueName: \"kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk\") pod \"console-7984696447-ftz9n\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.338900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.338795 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:26.486657 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.486626 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:19:26.489985 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:19:26.489961 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc9345f_a991_4133_83e1_fd920ec21a88.slice/crio-28feb1ab18e6cc6d62fa6ed72e1954f3f32f719986eff0b3b409670af3621021 WatchSource:0}: Error finding container 28feb1ab18e6cc6d62fa6ed72e1954f3f32f719986eff0b3b409670af3621021: Status 404 returned error can't find the container with id 28feb1ab18e6cc6d62fa6ed72e1954f3f32f719986eff0b3b409670af3621021 Jul 29 21:19:26.732389 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.732355 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7984696447-ftz9n" event={"ID":"7cc9345f-a991-4133-83e1-fd920ec21a88","Type":"ContainerStarted","Data":"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42"} Jul 29 21:19:26.732569 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.732427 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7984696447-ftz9n" event={"ID":"7cc9345f-a991-4133-83e1-fd920ec21a88","Type":"ContainerStarted","Data":"28feb1ab18e6cc6d62fa6ed72e1954f3f32f719986eff0b3b409670af3621021"} Jul 29 21:19:26.751164 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:26.751108 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7984696447-ftz9n" podStartSLOduration=0.75109281 podStartE2EDuration="751.09281ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:19:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:19:26.750412778 +0000 UTC m=+182.440844597" watchObservedRunningTime="2026-07-29 21:19:26.75109281 +0000 UTC m=+182.441524629" Jul 29 21:19:36.339010 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:36.338952 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:36.339633 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:36.339042 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:36.343804 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:36.343782 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:36.777492 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:36.777458 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:19:36.820009 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:19:36.819971 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:20:01.844579 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:01.844524 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8475b99f8b-jt79f" podUID="a7516792-a4a5-4bcf-a975-54e13d273166" containerName="console" containerID="cri-o://ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db" gracePeriod=15 Jul 29 21:20:02.087547 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.087523 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:20:02.169189 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169155 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169385 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169205 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169385 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169230 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169385 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169247 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzvl4\" (UniqueName: \"kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169385 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169371 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169584 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169421 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169584 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169457 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config\") pod \"a7516792-a4a5-4bcf-a975-54e13d273166\" (UID: \"a7516792-a4a5-4bcf-a975-54e13d273166\") " Jul 29 21:20:02.169773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169633 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:02.169839 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169790 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:02.169966 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.169911 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:02.170198 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.170041 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config" (OuterVolumeSpecName: "console-config") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:20:02.171645 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.171621 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:20:02.172102 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.172081 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:20:02.172169 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.172116 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4" (OuterVolumeSpecName: "kube-api-access-hzvl4") pod "a7516792-a4a5-4bcf-a975-54e13d273166" (UID: "a7516792-a4a5-4bcf-a975-54e13d273166"). InnerVolumeSpecName "kube-api-access-hzvl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:20:02.270210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270178 2527 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-trusted-ca-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270202 2527 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270214 2527 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-console-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270436 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270223 2527 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7516792-a4a5-4bcf-a975-54e13d273166-oauth-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270436 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270231 2527 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270436 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270240 2527 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7516792-a4a5-4bcf-a975-54e13d273166-console-oauth-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.270436 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.270248 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzvl4\" (UniqueName: \"kubernetes.io/projected/a7516792-a4a5-4bcf-a975-54e13d273166-kube-api-access-hzvl4\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:20:02.862669 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.862643 2527 generic.go:356] "Generic (PLEG): container finished" podID="a7516792-a4a5-4bcf-a975-54e13d273166" containerID="ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db" exitCode=0 Jul 29 21:20:02.863079 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.862685 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8475b99f8b-jt79f" Jul 29 21:20:02.863079 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.862724 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475b99f8b-jt79f" event={"ID":"a7516792-a4a5-4bcf-a975-54e13d273166","Type":"ContainerDied","Data":"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db"} Jul 29 21:20:02.863079 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.862782 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8475b99f8b-jt79f" event={"ID":"a7516792-a4a5-4bcf-a975-54e13d273166","Type":"ContainerDied","Data":"f99084437dc0743447264ac8ab3e8408c120b7edff6f1b8103657bc7d7943f67"} Jul 29 21:20:02.863079 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.862811 2527 scope.go:117] "RemoveContainer" containerID="ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db" Jul 29 21:20:02.871086 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.871070 2527 scope.go:117] "RemoveContainer" containerID="ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db" Jul 29 21:20:02.871331 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:20:02.871313 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db\": container with ID starting with ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db not found: ID does not exist" containerID="ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db" Jul 29 21:20:02.871384 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.871338 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db"} err="failed to get container status \"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db\": rpc error: code = NotFound desc = could not find container \"ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db\": container with ID starting with ca9bc45cdd94ad33b4485ede028d43c7820162d4feb08a46ac1d3ef9c3bbb1db not found: ID does not exist" Jul 29 21:20:02.878412 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.878388 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:20:02.881983 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:02.881962 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8475b99f8b-jt79f"] Jul 29 21:20:04.850880 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:04.850848 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7516792-a4a5-4bcf-a975-54e13d273166" path="/var/lib/kubelet/pods/a7516792-a4a5-4bcf-a975-54e13d273166/volumes" Jul 29 21:20:35.294617 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.294584 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:20:35.295190 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.294824 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7516792-a4a5-4bcf-a975-54e13d273166" containerName="console" Jul 29 21:20:35.295190 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.294834 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7516792-a4a5-4bcf-a975-54e13d273166" containerName="console" Jul 29 21:20:35.295190 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.294883 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7516792-a4a5-4bcf-a975-54e13d273166" containerName="console" Jul 29 21:20:35.298961 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.298916 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.308243 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.308212 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:20:35.324718 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324683 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324718 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324717 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4c2c\" (UniqueName: \"kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324746 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324815 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324847 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324876 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.324955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.324919 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426230 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426190 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426230 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426233 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426466 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426256 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4c2c\" (UniqueName: \"kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426466 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426287 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426466 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426320 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426466 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426347 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.426628 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.426523 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.427036 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.427007 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.427151 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.427141 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.427209 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.427153 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.427429 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.427407 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.428970 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.428949 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.429067 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.429024 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.434181 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.434161 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4c2c\" (UniqueName: \"kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c\") pod \"console-79f69d98b9-rp82b\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.609459 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.609423 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:35.746508 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.746483 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:20:35.748850 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:20:35.748820 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0239014_0e17_49d6_92e3_0ff923388ae2.slice/crio-42a58a2d28e2d69100ba6044761871740ceb068c0b66afe19279ddb1361cf55e WatchSource:0}: Error finding container 42a58a2d28e2d69100ba6044761871740ceb068c0b66afe19279ddb1361cf55e: Status 404 returned error can't find the container with id 42a58a2d28e2d69100ba6044761871740ceb068c0b66afe19279ddb1361cf55e Jul 29 21:20:35.978410 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.978322 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f69d98b9-rp82b" event={"ID":"f0239014-0e17-49d6-92e3-0ff923388ae2","Type":"ContainerStarted","Data":"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6"} Jul 29 21:20:35.978410 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.978372 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f69d98b9-rp82b" event={"ID":"f0239014-0e17-49d6-92e3-0ff923388ae2","Type":"ContainerStarted","Data":"42a58a2d28e2d69100ba6044761871740ceb068c0b66afe19279ddb1361cf55e"} Jul 29 21:20:35.996437 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:35.996248 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79f69d98b9-rp82b" podStartSLOduration=0.996225888 podStartE2EDuration="996.225888ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:20:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:20:35.995449991 +0000 UTC m=+251.685881815" watchObservedRunningTime="2026-07-29 21:20:35.996225888 +0000 UTC m=+251.686657708" Jul 29 21:20:45.609691 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:45.609650 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:45.610178 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:45.609720 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:45.614467 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:45.614440 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:46.017630 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:46.017521 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:20:46.065826 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:20:46.065793 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:21:11.089129 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.089088 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7984696447-ftz9n" podUID="7cc9345f-a991-4133-83e1-fd920ec21a88" containerName="console" containerID="cri-o://b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42" gracePeriod=15 Jul 29 21:21:11.334189 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.334163 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:21:11.401886 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.401847 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.401915 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.401965 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.401994 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402251 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402154 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402251 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402210 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402251 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402246 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkkhk\" (UniqueName: \"kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk\") pod \"7cc9345f-a991-4133-83e1-fd920ec21a88\" (UID: \"7cc9345f-a991-4133-83e1-fd920ec21a88\") " Jul 29 21:21:11.402400 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402378 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config" (OuterVolumeSpecName: "console-config") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:21:11.402454 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402390 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:21:11.402506 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402483 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca" (OuterVolumeSpecName: "service-ca") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:21:11.402563 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402536 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:21:11.402618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402565 2527 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-console-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.402618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402580 2527 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-trusted-ca-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.402618 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.402593 2527 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.404259 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.404237 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:21:11.404397 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.404375 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:21:11.404493 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.404472 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk" (OuterVolumeSpecName: "kube-api-access-kkkhk") pod "7cc9345f-a991-4133-83e1-fd920ec21a88" (UID: "7cc9345f-a991-4133-83e1-fd920ec21a88"). InnerVolumeSpecName "kube-api-access-kkkhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:21:11.503668 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.503628 2527 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-oauth-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.503668 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.503661 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkkhk\" (UniqueName: \"kubernetes.io/projected/7cc9345f-a991-4133-83e1-fd920ec21a88-kube-api-access-kkkhk\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.503668 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.503675 2527 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/7cc9345f-a991-4133-83e1-fd920ec21a88-console-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:11.503881 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:11.503686 2527 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/7cc9345f-a991-4133-83e1-fd920ec21a88-oauth-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:12.104579 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.104541 2527 generic.go:356] "Generic (PLEG): container finished" podID="7cc9345f-a991-4133-83e1-fd920ec21a88" containerID="b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42" exitCode=0 Jul 29 21:21:12.105024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.104589 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7984696447-ftz9n" Jul 29 21:21:12.105024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.104630 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7984696447-ftz9n" event={"ID":"7cc9345f-a991-4133-83e1-fd920ec21a88","Type":"ContainerDied","Data":"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42"} Jul 29 21:21:12.105024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.104681 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7984696447-ftz9n" event={"ID":"7cc9345f-a991-4133-83e1-fd920ec21a88","Type":"ContainerDied","Data":"28feb1ab18e6cc6d62fa6ed72e1954f3f32f719986eff0b3b409670af3621021"} Jul 29 21:21:12.105024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.104718 2527 scope.go:117] "RemoveContainer" containerID="b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42" Jul 29 21:21:12.113030 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.113013 2527 scope.go:117] "RemoveContainer" containerID="b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42" Jul 29 21:21:12.113300 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:21:12.113281 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42\": container with ID starting with b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42 not found: ID does not exist" containerID="b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42" Jul 29 21:21:12.113342 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.113310 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42"} err="failed to get container status \"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42\": rpc error: code = NotFound desc = could not find container \"b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42\": container with ID starting with b20618caa0c79951bd651cf34c4ab851057d3fe5aec809e28e3fd5c812539c42 not found: ID does not exist" Jul 29 21:21:12.126260 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.126235 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:21:12.129333 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.129309 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7984696447-ftz9n"] Jul 29 21:21:12.851563 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:12.851534 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc9345f-a991-4133-83e1-fd920ec21a88" path="/var/lib/kubelet/pods/7cc9345f-a991-4133-83e1-fd920ec21a88/volumes" Jul 29 21:21:24.726327 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:24.726300 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:21:24.733218 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:24.733192 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:21:24.735964 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:24.735907 2527 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 21:21:24.736641 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:24.736611 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:21:24.744460 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:24.744437 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:21:35.113907 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.113872 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58"] Jul 29 21:21:35.116194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.114136 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc9345f-a991-4133-83e1-fd920ec21a88" containerName="console" Jul 29 21:21:35.116194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.114147 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc9345f-a991-4133-83e1-fd920ec21a88" containerName="console" Jul 29 21:21:35.116194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.114190 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc9345f-a991-4133-83e1-fd920ec21a88" containerName="console" Jul 29 21:21:35.117120 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.117105 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.119801 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.119774 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 21:21:35.119949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.119779 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 21:21:35.120024 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.119998 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ff6wj\"" Jul 29 21:21:35.126740 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.126715 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58"] Jul 29 21:21:35.285616 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.285580 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.285808 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.285623 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.285808 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.285646 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbv64\" (UniqueName: \"kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.386279 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.386178 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.386279 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.386230 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.386279 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.386263 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbv64\" (UniqueName: \"kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.386595 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.386574 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.386655 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.386591 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.396735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.396706 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbv64\" (UniqueName: \"kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.425623 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.425588 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:35.562549 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.562400 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58"] Jul 29 21:21:35.565287 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:21:35.565258 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c2b7115_32e2_498f_acb6_2d81fac8ca54.slice/crio-75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28 WatchSource:0}: Error finding container 75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28: Status 404 returned error can't find the container with id 75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28 Jul 29 21:21:35.567460 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:35.567441 2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:21:36.189187 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:36.189148 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" event={"ID":"2c2b7115-32e2-498f-acb6-2d81fac8ca54","Type":"ContainerStarted","Data":"75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28"} Jul 29 21:21:44.221399 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:44.221354 2527 generic.go:356] "Generic (PLEG): container finished" podID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerID="4845bb2aed90ef17c1abf7d6f620837e81cfa765611cb3ac90d11aaaaf31a843" exitCode=0 Jul 29 21:21:44.221799 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:44.221406 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" event={"ID":"2c2b7115-32e2-498f-acb6-2d81fac8ca54","Type":"ContainerDied","Data":"4845bb2aed90ef17c1abf7d6f620837e81cfa765611cb3ac90d11aaaaf31a843"} Jul 29 21:21:47.235774 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:47.235740 2527 generic.go:356] "Generic (PLEG): container finished" podID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerID="22238b18f33833080621d0f1d56ec9914de4fd15286958d92567abfabd78b22c" exitCode=0 Jul 29 21:21:47.236163 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:47.235820 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" event={"ID":"2c2b7115-32e2-498f-acb6-2d81fac8ca54","Type":"ContainerDied","Data":"22238b18f33833080621d0f1d56ec9914de4fd15286958d92567abfabd78b22c"} Jul 29 21:21:55.269262 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:55.269222 2527 generic.go:356] "Generic (PLEG): container finished" podID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerID="e51b4c8c438ccd08f4dd3efb2da2ea0ebb47317e243326b716fec1b8a1155c69" exitCode=0 Jul 29 21:21:55.269666 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:55.269301 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" event={"ID":"2c2b7115-32e2-498f-acb6-2d81fac8ca54","Type":"ContainerDied","Data":"e51b4c8c438ccd08f4dd3efb2da2ea0ebb47317e243326b716fec1b8a1155c69"} Jul 29 21:21:56.395801 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.395775 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:56.466073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.466032 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle\") pod \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " Jul 29 21:21:56.466073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.466080 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbv64\" (UniqueName: \"kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64\") pod \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " Jul 29 21:21:56.466292 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.466128 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util\") pod \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\" (UID: \"2c2b7115-32e2-498f-acb6-2d81fac8ca54\") " Jul 29 21:21:56.466696 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.466667 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle" (OuterVolumeSpecName: "bundle") pod "2c2b7115-32e2-498f-acb6-2d81fac8ca54" (UID: "2c2b7115-32e2-498f-acb6-2d81fac8ca54"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:21:56.468451 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.468426 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64" (OuterVolumeSpecName: "kube-api-access-vbv64") pod "2c2b7115-32e2-498f-acb6-2d81fac8ca54" (UID: "2c2b7115-32e2-498f-acb6-2d81fac8ca54"). InnerVolumeSpecName "kube-api-access-vbv64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:21:56.471331 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.471311 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util" (OuterVolumeSpecName: "util") pod "2c2b7115-32e2-498f-acb6-2d81fac8ca54" (UID: "2c2b7115-32e2-498f-acb6-2d81fac8ca54"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:21:56.566602 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.566508 2527 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-util\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:56.566602 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.566541 2527 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c2b7115-32e2-498f-acb6-2d81fac8ca54-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:56.566602 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:56.566555 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbv64\" (UniqueName: \"kubernetes.io/projected/2c2b7115-32e2-498f-acb6-2d81fac8ca54-kube-api-access-vbv64\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:21:57.278118 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:57.278039 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" Jul 29 21:21:57.278242 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:57.278038 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675402dd58" event={"ID":"2c2b7115-32e2-498f-acb6-2d81fac8ca54","Type":"ContainerDied","Data":"75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28"} Jul 29 21:21:57.278242 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:21:57.278198 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75cdc0d310047ce97cbbb8b66743030175157030b3cf14cfa9233bbf23653d28" Jul 29 21:22:08.577613 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577576 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk"] Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577827 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="util" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577840 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="util" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577858 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="pull" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577863 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="pull" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577869 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="extract" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577874 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="extract" Jul 29 21:22:08.578073 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.577918 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c2b7115-32e2-498f-acb6-2d81fac8ca54" containerName="extract" Jul 29 21:22:08.583014 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.582994 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.587196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.587159 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ff6wj\"" Jul 29 21:22:08.587196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.587169 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 21:22:08.587365 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.587230 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 21:22:08.617510 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.617484 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk"] Jul 29 21:22:08.657058 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.657019 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnql\" (UniqueName: \"kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.657236 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.657071 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.657236 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.657130 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.757548 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.757506 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnql\" (UniqueName: \"kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.757714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.757569 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.757714 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.757594 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.757995 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.757961 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.758210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.758186 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.801919 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.801885 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnql\" (UniqueName: \"kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:08.892320 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:08.892271 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:09.040316 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:09.040282 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk"] Jul 29 21:22:09.044058 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:22:09.044028 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71dd6f75_017f_4775_bb6b_5cfc97b272fc.slice/crio-246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2 WatchSource:0}: Error finding container 246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2: Status 404 returned error can't find the container with id 246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2 Jul 29 21:22:09.321006 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:09.320969 2527 generic.go:356] "Generic (PLEG): container finished" podID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerID="b1013f3cc364c1ae1659c1fd8f800bdcab4772e7df6e8dbc11d679e981961300" exitCode=0 Jul 29 21:22:09.321183 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:09.321014 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" event={"ID":"71dd6f75-017f-4775-bb6b-5cfc97b272fc","Type":"ContainerDied","Data":"b1013f3cc364c1ae1659c1fd8f800bdcab4772e7df6e8dbc11d679e981961300"} Jul 29 21:22:09.321183 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:09.321073 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" event={"ID":"71dd6f75-017f-4775-bb6b-5cfc97b272fc","Type":"ContainerStarted","Data":"246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2"} Jul 29 21:22:11.331403 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:11.331372 2527 generic.go:356] "Generic (PLEG): container finished" podID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerID="0dd8ad6f3c28bd17b01cb6a5c5bd454d3625c5f3527d78c90083f951590cc8f7" exitCode=0 Jul 29 21:22:11.331879 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:11.331413 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" event={"ID":"71dd6f75-017f-4775-bb6b-5cfc97b272fc","Type":"ContainerDied","Data":"0dd8ad6f3c28bd17b01cb6a5c5bd454d3625c5f3527d78c90083f951590cc8f7"} Jul 29 21:22:12.341200 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:12.341159 2527 generic.go:356] "Generic (PLEG): container finished" podID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerID="4c873f309451d12b2869502c50dfd0acd2b55f73e5af39b7f0151f8f99eae6a6" exitCode=0 Jul 29 21:22:12.341200 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:12.341195 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" event={"ID":"71dd6f75-017f-4775-bb6b-5cfc97b272fc","Type":"ContainerDied","Data":"4c873f309451d12b2869502c50dfd0acd2b55f73e5af39b7f0151f8f99eae6a6"} Jul 29 21:22:13.472536 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.472512 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:13.599011 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.598878 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle\") pod \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " Jul 29 21:22:13.599011 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.598960 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pnql\" (UniqueName: \"kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql\") pod \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " Jul 29 21:22:13.599252 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.599037 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util\") pod \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\" (UID: \"71dd6f75-017f-4775-bb6b-5cfc97b272fc\") " Jul 29 21:22:13.599310 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.599286 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle" (OuterVolumeSpecName: "bundle") pod "71dd6f75-017f-4775-bb6b-5cfc97b272fc" (UID: "71dd6f75-017f-4775-bb6b-5cfc97b272fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:22:13.601257 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.601230 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql" (OuterVolumeSpecName: "kube-api-access-5pnql") pod "71dd6f75-017f-4775-bb6b-5cfc97b272fc" (UID: "71dd6f75-017f-4775-bb6b-5cfc97b272fc"). InnerVolumeSpecName "kube-api-access-5pnql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:22:13.607571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.607524 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util" (OuterVolumeSpecName: "util") pod "71dd6f75-017f-4775-bb6b-5cfc97b272fc" (UID: "71dd6f75-017f-4775-bb6b-5cfc97b272fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:22:13.700182 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.700127 2527 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:13.700182 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.700173 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnql\" (UniqueName: \"kubernetes.io/projected/71dd6f75-017f-4775-bb6b-5cfc97b272fc-kube-api-access-5pnql\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:13.700182 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:13.700184 2527 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/71dd6f75-017f-4775-bb6b-5cfc97b272fc-util\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:14.350734 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:14.350699 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" event={"ID":"71dd6f75-017f-4775-bb6b-5cfc97b272fc","Type":"ContainerDied","Data":"246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2"} Jul 29 21:22:14.350734 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:14.350737 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ft2mmk" Jul 29 21:22:14.350968 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:14.350751 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="246a8cc407500cf70db945fde0c7e297e4cc4fd68189bb91570acc2f26ab1aa2" Jul 29 21:22:20.025883 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.025802 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr"] Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026052 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="extract" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026063 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="extract" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026075 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="util" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026081 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="util" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026094 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="pull" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026102 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="pull" Jul 29 21:22:20.026267 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.026150 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="71dd6f75-017f-4775-bb6b-5cfc97b272fc" containerName="extract" Jul 29 21:22:20.028808 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.028792 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.031074 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.031051 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 21:22:20.031196 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.031157 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 21:22:20.031673 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.031659 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-kn5pr\"" Jul 29 21:22:20.038271 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.038249 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr"] Jul 29 21:22:20.153865 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.153827 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1344f346-30eb-45e7-930b-7fd42e7e7a2c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.153865 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.153865 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pd6p\" (UniqueName: \"kubernetes.io/projected/1344f346-30eb-45e7-930b-7fd42e7e7a2c-kube-api-access-4pd6p\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.254442 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.254403 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1344f346-30eb-45e7-930b-7fd42e7e7a2c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.254442 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.254443 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pd6p\" (UniqueName: \"kubernetes.io/projected/1344f346-30eb-45e7-930b-7fd42e7e7a2c-kube-api-access-4pd6p\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.254785 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.254764 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1344f346-30eb-45e7-930b-7fd42e7e7a2c-tmp\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.263214 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.263182 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pd6p\" (UniqueName: \"kubernetes.io/projected/1344f346-30eb-45e7-930b-7fd42e7e7a2c-kube-api-access-4pd6p\") pod \"openshift-lws-operator-fd8ccff4c-dq2vr\" (UID: \"1344f346-30eb-45e7-930b-7fd42e7e7a2c\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.338279 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.338184 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" Jul 29 21:22:20.473464 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:20.473434 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr"] Jul 29 21:22:20.476433 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:22:20.476407 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1344f346_30eb_45e7_930b_7fd42e7e7a2c.slice/crio-a8451dc47a4481a9a7402f5148be450db4590d443c13829e527be50143fdf40c WatchSource:0}: Error finding container a8451dc47a4481a9a7402f5148be450db4590d443c13829e527be50143fdf40c: Status 404 returned error can't find the container with id a8451dc47a4481a9a7402f5148be450db4590d443c13829e527be50143fdf40c Jul 29 21:22:21.378014 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:21.377980 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" event={"ID":"1344f346-30eb-45e7-930b-7fd42e7e7a2c","Type":"ContainerStarted","Data":"a8451dc47a4481a9a7402f5148be450db4590d443c13829e527be50143fdf40c"} Jul 29 21:22:23.387264 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:23.387225 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" event={"ID":"1344f346-30eb-45e7-930b-7fd42e7e7a2c","Type":"ContainerStarted","Data":"52946813b4384a4209f4640ddf21ddbf79dc3e14421b1e005a22e79a774dd609"} Jul 29 21:22:23.409060 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:23.409001 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-dq2vr" podStartSLOduration=1.461797872 podStartE2EDuration="3.408979155s" totalImagesPullingTime="1.947181283s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:23.407240074 +0000 UTC m=+359.097671892" watchObservedRunningTime="2026-07-29 21:22:23.408979155 +0000 UTC m=+359.099410975" Jul 29 21:22:26.147806 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.147765 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k"] Jul 29 21:22:26.152015 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.151997 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.154420 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.154397 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 29 21:22:26.155097 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.155077 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 29 21:22:26.155195 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.155119 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ff6wj\"" Jul 29 21:22:26.159706 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.159685 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k"] Jul 29 21:22:26.302749 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.302716 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hfr9\" (UniqueName: \"kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.302915 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.302775 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.302915 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.302866 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.403469 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.403375 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.403469 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.403441 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.403697 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.403481 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hfr9\" (UniqueName: \"kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.403754 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.403727 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.403802 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.403762 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.411636 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.411609 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hfr9\" (UniqueName: \"kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.462050 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.462018 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:26.590029 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:26.589997 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k"] Jul 29 21:22:26.592082 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:22:26.592053 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f77a65f_899b_4c40_b519_6b0a10c11cb1.slice/crio-1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab WatchSource:0}: Error finding container 1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab: Status 404 returned error can't find the container with id 1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab Jul 29 21:22:27.402823 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:27.402793 2527 generic.go:356] "Generic (PLEG): container finished" podID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerID="237971c31cedf6c9d8da74940ce0907c81ed278f3a09301d1dcdefda68bd4c40" exitCode=0 Jul 29 21:22:27.403240 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:27.402852 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" event={"ID":"2f77a65f-899b-4c40-b519-6b0a10c11cb1","Type":"ContainerDied","Data":"237971c31cedf6c9d8da74940ce0907c81ed278f3a09301d1dcdefda68bd4c40"} Jul 29 21:22:27.403240 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:27.402895 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" event={"ID":"2f77a65f-899b-4c40-b519-6b0a10c11cb1","Type":"ContainerStarted","Data":"1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab"} Jul 29 21:22:28.408635 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:28.408595 2527 generic.go:356] "Generic (PLEG): container finished" podID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerID="bee8cb60e974cb5e0640bde8a8f9b6ff7f22b6acb6151075a73e5ea3090f0e8e" exitCode=0 Jul 29 21:22:28.409044 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:28.408654 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" event={"ID":"2f77a65f-899b-4c40-b519-6b0a10c11cb1","Type":"ContainerDied","Data":"bee8cb60e974cb5e0640bde8a8f9b6ff7f22b6acb6151075a73e5ea3090f0e8e"} Jul 29 21:22:29.414880 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:29.414847 2527 generic.go:356] "Generic (PLEG): container finished" podID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerID="72408c76318ad71d68bf840174b86a7fa41131008aa472babff3e56aa30c9850" exitCode=0 Jul 29 21:22:29.415295 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:29.414957 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" event={"ID":"2f77a65f-899b-4c40-b519-6b0a10c11cb1","Type":"ContainerDied","Data":"72408c76318ad71d68bf840174b86a7fa41131008aa472babff3e56aa30c9850"} Jul 29 21:22:30.542447 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.542422 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:30.639394 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.639356 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util\") pod \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " Jul 29 21:22:30.639593 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.639434 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle\") pod \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " Jul 29 21:22:30.639593 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.639471 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hfr9\" (UniqueName: \"kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9\") pod \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\" (UID: \"2f77a65f-899b-4c40-b519-6b0a10c11cb1\") " Jul 29 21:22:30.640212 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.640186 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle" (OuterVolumeSpecName: "bundle") pod "2f77a65f-899b-4c40-b519-6b0a10c11cb1" (UID: "2f77a65f-899b-4c40-b519-6b0a10c11cb1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:22:30.641664 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.641637 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9" (OuterVolumeSpecName: "kube-api-access-5hfr9") pod "2f77a65f-899b-4c40-b519-6b0a10c11cb1" (UID: "2f77a65f-899b-4c40-b519-6b0a10c11cb1"). InnerVolumeSpecName "kube-api-access-5hfr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:22:30.645793 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.645772 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util" (OuterVolumeSpecName: "util") pod "2f77a65f-899b-4c40-b519-6b0a10c11cb1" (UID: "2f77a65f-899b-4c40-b519-6b0a10c11cb1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 21:22:30.741035 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.740908 2527 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:30.741035 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.740983 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hfr9\" (UniqueName: \"kubernetes.io/projected/2f77a65f-899b-4c40-b519-6b0a10c11cb1-kube-api-access-5hfr9\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:30.741035 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:30.740993 2527 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f77a65f-899b-4c40-b519-6b0a10c11cb1-util\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:22:31.424364 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:31.424329 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" event={"ID":"2f77a65f-899b-4c40-b519-6b0a10c11cb1","Type":"ContainerDied","Data":"1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab"} Jul 29 21:22:31.424364 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:31.424347 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afwg54k" Jul 29 21:22:31.424573 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:31.424383 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fbc91a193052ea69e2e918b5ced1c2b3de77e6bcaf94b34d8eb7fb70c78beab" Jul 29 21:22:43.137684 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137653 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4"] Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137900 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="pull" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137909 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="pull" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137951 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="util" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137958 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="util" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137966 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="extract" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.137973 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="extract" Jul 29 21:22:43.138076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.138036 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f77a65f-899b-4c40-b519-6b0a10c11cb1" containerName="extract" Jul 29 21:22:43.142092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.142069 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.144387 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.144358 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 21:22:43.144524 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.144362 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 21:22:43.144524 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.144412 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 21:22:43.144524 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.144435 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-sk4g6\"" Jul 29 21:22:43.144823 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.144803 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 21:22:43.158525 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.158499 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4"] Jul 29 21:22:43.238553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.238516 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.238553 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.238554 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.238755 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.238575 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.238755 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.238594 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.238755 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.238697 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhx6c\" (UniqueName: \"kubernetes.io/projected/e05fe207-19c8-471e-8c6e-14074c22b864-kube-api-access-jhx6c\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.339941 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.339905 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhx6c\" (UniqueName: \"kubernetes.io/projected/e05fe207-19c8-471e-8c6e-14074c22b864-kube-api-access-jhx6c\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.340114 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.339971 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.340114 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.339995 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.340114 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.340013 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.340114 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.340034 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.340515 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.340492 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-manifests\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.342468 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.342449 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e05fe207-19c8-471e-8c6e-14074c22b864-tmp\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.342564 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.342509 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.342626 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.342597 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e05fe207-19c8-471e-8c6e-14074c22b864-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.361312 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.361277 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhx6c\" (UniqueName: \"kubernetes.io/projected/e05fe207-19c8-471e-8c6e-14074c22b864-kube-api-access-jhx6c\") pod \"opendatahub-operator-controller-manager-6b55745745-mnxd4\" (UID: \"e05fe207-19c8-471e-8c6e-14074c22b864\") " pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.452603 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.452507 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:43.590131 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:43.590106 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4"] Jul 29 21:22:43.592705 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:22:43.592665 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05fe207_19c8_471e_8c6e_14074c22b864.slice/crio-7312d721f335454ac3d050aa1c09e30cfcd8eafa3975791720c3a65d13f27de0 WatchSource:0}: Error finding container 7312d721f335454ac3d050aa1c09e30cfcd8eafa3975791720c3a65d13f27de0: Status 404 returned error can't find the container with id 7312d721f335454ac3d050aa1c09e30cfcd8eafa3975791720c3a65d13f27de0 Jul 29 21:22:44.479797 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:44.479753 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" event={"ID":"e05fe207-19c8-471e-8c6e-14074c22b864","Type":"ContainerStarted","Data":"7312d721f335454ac3d050aa1c09e30cfcd8eafa3975791720c3a65d13f27de0"} Jul 29 21:22:47.495577 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:47.495534 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" event={"ID":"e05fe207-19c8-471e-8c6e-14074c22b864","Type":"ContainerStarted","Data":"16cceea9010093a04efe31514166c5bbdfb2009ce61c3281587d7b86a47ad39a"} Jul 29 21:22:48.500905 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:48.500870 2527 generic.go:356] "Generic (PLEG): container finished" podID="e05fe207-19c8-471e-8c6e-14074c22b864" containerID="16cceea9010093a04efe31514166c5bbdfb2009ce61c3281587d7b86a47ad39a" exitCode=0 Jul 29 21:22:48.501272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:48.500957 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" event={"ID":"e05fe207-19c8-471e-8c6e-14074c22b864","Type":"ContainerDied","Data":"16cceea9010093a04efe31514166c5bbdfb2009ce61c3281587d7b86a47ad39a"} Jul 29 21:22:49.507200 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:49.507164 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" event={"ID":"e05fe207-19c8-471e-8c6e-14074c22b864","Type":"ContainerStarted","Data":"75bdec5af6bcc255337aecb26270399c1ea4ecc1cdadb9797fbdc8bbd54c9257"} Jul 29 21:22:49.507656 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:49.507271 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:22:49.529379 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:22:49.529325 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" podStartSLOduration=1.550261109 podStartE2EDuration="6.529307919s" totalImagesPullingTime="3.967405966s" totalInitContainerRuntime="1.011640844s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:22:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:22:49.527883451 +0000 UTC m=+385.218315307" watchObservedRunningTime="2026-07-29 21:22:49.529307919 +0000 UTC m=+385.219739737" Jul 29 21:23:00.514947 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:00.514893 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b55745745-mnxd4" Jul 29 21:23:08.645756 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.645717 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-787cd99557-2s4t2"] Jul 29 21:23:08.653911 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.653890 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.656696 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.656676 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 21:23:08.657632 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.657614 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 21:23:08.657706 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.657616 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xxzjf\"" Jul 29 21:23:08.661678 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.661641 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-787cd99557-2s4t2"] Jul 29 21:23:08.738878 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.738841 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d358801-09e8-4796-bd37-812f2ad9ae85-tls-certs\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.739081 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.738898 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d358801-09e8-4796-bd37-812f2ad9ae85-tmp\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.739081 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.738952 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h24gq\" (UniqueName: \"kubernetes.io/projected/8d358801-09e8-4796-bd37-812f2ad9ae85-kube-api-access-h24gq\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.840072 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.840028 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d358801-09e8-4796-bd37-812f2ad9ae85-tls-certs\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.840256 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.840095 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d358801-09e8-4796-bd37-812f2ad9ae85-tmp\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.840256 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.840124 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h24gq\" (UniqueName: \"kubernetes.io/projected/8d358801-09e8-4796-bd37-812f2ad9ae85-kube-api-access-h24gq\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.842525 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.842494 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d358801-09e8-4796-bd37-812f2ad9ae85-tmp\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.842659 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.842619 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d358801-09e8-4796-bd37-812f2ad9ae85-tls-certs\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.850528 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.850505 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h24gq\" (UniqueName: \"kubernetes.io/projected/8d358801-09e8-4796-bd37-812f2ad9ae85-kube-api-access-h24gq\") pod \"kube-auth-proxy-787cd99557-2s4t2\" (UID: \"8d358801-09e8-4796-bd37-812f2ad9ae85\") " pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:08.964787 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:08.964697 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" Jul 29 21:23:09.099665 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:09.099577 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-787cd99557-2s4t2"] Jul 29 21:23:09.102388 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:23:09.102360 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d358801_09e8_4796_bd37_812f2ad9ae85.slice/crio-e577f03a9e727a892fa0bb5cc1ca95c190c7d757ceed1e12937a83a876194437 WatchSource:0}: Error finding container e577f03a9e727a892fa0bb5cc1ca95c190c7d757ceed1e12937a83a876194437: Status 404 returned error can't find the container with id e577f03a9e727a892fa0bb5cc1ca95c190c7d757ceed1e12937a83a876194437 Jul 29 21:23:09.592343 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:09.592308 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" event={"ID":"8d358801-09e8-4796-bd37-812f2ad9ae85","Type":"ContainerStarted","Data":"e577f03a9e727a892fa0bb5cc1ca95c190c7d757ceed1e12937a83a876194437"} Jul 29 21:23:11.683427 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.683384 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk"] Jul 29 21:23:11.687295 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.687267 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.689866 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.689808 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 21:23:11.689866 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.689809 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 21:23:11.690275 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.690255 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 21:23:11.690563 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.690544 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 21:23:11.693294 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.693275 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vkt9s\"" Jul 29 21:23:11.704832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.704804 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk"] Jul 29 21:23:11.767610 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767579 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36096a1e-645c-426a-a38a-a287073b1b50-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767625 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnphz\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-kube-api-access-lnphz\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767679 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767704 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767837 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767751 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767837 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767781 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.767907 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.767894 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868708 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868672 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868708 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868714 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868756 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868776 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36096a1e-645c-426a-a38a-a287073b1b50-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868795 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnphz\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-kube-api-access-lnphz\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868812 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.868999 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.868835 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.869392 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.869360 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.871262 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.871239 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/36096a1e-645c-426a-a38a-a287073b1b50-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.871372 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.871304 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.871656 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.871636 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.871735 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.871706 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/36096a1e-645c-426a-a38a-a287073b1b50-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.886398 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.886365 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.886606 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.886581 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnphz\" (UniqueName: \"kubernetes.io/projected/36096a1e-645c-426a-a38a-a287073b1b50-kube-api-access-lnphz\") pod \"istiod-openshift-gateway-5677ccb49b-md5bk\" (UID: \"36096a1e-645c-426a-a38a-a287073b1b50\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:11.999591 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:11.999494 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:12.080098 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.080066 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6"] Jul 29 21:23:12.084054 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.083984 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.088040 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.088012 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 29 21:23:12.092177 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.092136 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 29 21:23:12.098070 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.098019 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 29 21:23:12.098351 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.098336 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-2k4ht\"" Jul 29 21:23:12.169866 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.169837 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk"] Jul 29 21:23:12.171796 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.171767 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8dcf\" (UniqueName: \"kubernetes.io/projected/e975faba-e82e-4892-bfe0-26bbd9fdc237-kube-api-access-w8dcf\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.171903 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.171825 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.171903 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.171861 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e975faba-e82e-4892-bfe0-26bbd9fdc237-manager-config\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.171903 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.171879 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.172453 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:23:12.172428 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36096a1e_645c_426a_a38a_a287073b1b50.slice/crio-2766ccaab63dea0e1510c6bea7e830d36b1fefa1c36f52f542bc6b6ab144c3cb WatchSource:0}: Error finding container 2766ccaab63dea0e1510c6bea7e830d36b1fefa1c36f52f542bc6b6ab144c3cb: Status 404 returned error can't find the container with id 2766ccaab63dea0e1510c6bea7e830d36b1fefa1c36f52f542bc6b6ab144c3cb Jul 29 21:23:12.180047 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.180027 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6"] Jul 29 21:23:12.272584 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.272502 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.272584 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.272544 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e975faba-e82e-4892-bfe0-26bbd9fdc237-manager-config\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.272584 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.272563 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.272846 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.272614 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8dcf\" (UniqueName: \"kubernetes.io/projected/e975faba-e82e-4892-bfe0-26bbd9fdc237-kube-api-access-w8dcf\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.273319 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.273256 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/e975faba-e82e-4892-bfe0-26bbd9fdc237-manager-config\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.275194 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.275172 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-metrics-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.275274 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.275226 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e975faba-e82e-4892-bfe0-26bbd9fdc237-cert\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.281786 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.281768 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8dcf\" (UniqueName: \"kubernetes.io/projected/e975faba-e82e-4892-bfe0-26bbd9fdc237-kube-api-access-w8dcf\") pod \"lws-controller-manager-5999ff5f6f-qg5p6\" (UID: \"e975faba-e82e-4892-bfe0-26bbd9fdc237\") " pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.394649 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.394613 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:12.530905 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.530734 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6"] Jul 29 21:23:12.533497 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:23:12.533466 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode975faba_e82e_4892_bfe0_26bbd9fdc237.slice/crio-2296aab13ef44016f78ac8af0bf8d551204aa9e9af0a107788a1dd736f665a41 WatchSource:0}: Error finding container 2296aab13ef44016f78ac8af0bf8d551204aa9e9af0a107788a1dd736f665a41: Status 404 returned error can't find the container with id 2296aab13ef44016f78ac8af0bf8d551204aa9e9af0a107788a1dd736f665a41 Jul 29 21:23:12.606885 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.606853 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" event={"ID":"8d358801-09e8-4796-bd37-812f2ad9ae85","Type":"ContainerStarted","Data":"00801e627060a0857cb611ca3cb5e431350d3b74fafc00b3a757798aea60fd99"} Jul 29 21:23:12.608066 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.608038 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" event={"ID":"e975faba-e82e-4892-bfe0-26bbd9fdc237","Type":"ContainerStarted","Data":"2296aab13ef44016f78ac8af0bf8d551204aa9e9af0a107788a1dd736f665a41"} Jul 29 21:23:12.609173 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.609153 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" event={"ID":"36096a1e-645c-426a-a38a-a287073b1b50","Type":"ContainerStarted","Data":"2766ccaab63dea0e1510c6bea7e830d36b1fefa1c36f52f542bc6b6ab144c3cb"} Jul 29 21:23:12.625157 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:12.625115 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-787cd99557-2s4t2" podStartSLOduration=1.966417262 podStartE2EDuration="4.625100788s" totalImagesPullingTime="2.658683526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:12.624131119 +0000 UTC m=+408.314562936" watchObservedRunningTime="2026-07-29 21:23:12.625100788 +0000 UTC m=+408.315532607" Jul 29 21:23:14.620906 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:14.620875 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" event={"ID":"e975faba-e82e-4892-bfe0-26bbd9fdc237","Type":"ContainerStarted","Data":"986ec2872b8ead55960654e8b90f8be625d550ff1aa58953149f8d12b27691bc"} Jul 29 21:23:14.621416 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:14.621002 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:23:14.652328 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:14.652267 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" podStartSLOduration=1.123806459 podStartE2EDuration="2.65224663s" totalImagesPullingTime="1.528440171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:14.650706117 +0000 UTC m=+410.341137935" watchObservedRunningTime="2026-07-29 21:23:14.65224663 +0000 UTC m=+410.342678447" Jul 29 21:23:19.874201 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:19.874153 2527 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:19.874486 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:19.874235 2527 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 21:23:20.648371 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:20.648342 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" event={"ID":"36096a1e-645c-426a-a38a-a287073b1b50","Type":"ContainerStarted","Data":"6e5cc54b26a90323eadb44622bcf29b26ee16b4476aa018567c97c3b5efafbef"} Jul 29 21:23:20.648593 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:20.648499 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:20.649897 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:20.649870 2527 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-md5bk container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 21:23:20.650028 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:20.649916 2527 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" podUID="36096a1e-645c-426a-a38a-a287073b1b50" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 21:23:20.693144 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:20.693093 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" podStartSLOduration=1.993615787 podStartE2EDuration="9.693056983s" totalImagesPullingTime="7.699441196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:23:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:23:20.692039171 +0000 UTC m=+416.382470989" watchObservedRunningTime="2026-07-29 21:23:20.693056983 +0000 UTC m=+416.383488801" Jul 29 21:23:21.653233 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:21.653206 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-md5bk" Jul 29 21:23:25.627953 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:23:25.627898 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5999ff5f6f-qg5p6" Jul 29 21:24:45.379867 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:24:45.379833 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:25:10.401529 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.401479 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79f69d98b9-rp82b" podUID="f0239014-0e17-49d6-92e3-0ff923388ae2" containerName="console" containerID="cri-o://a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6" gracePeriod=15 Jul 29 21:25:10.643331 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.643304 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:25:10.773433 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773337 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773436 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773474 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773512 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773539 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4c2c\" (UniqueName: \"kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773566 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773614 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773588 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca\") pod \"f0239014-0e17-49d6-92e3-0ff923388ae2\" (UID: \"f0239014-0e17-49d6-92e3-0ff923388ae2\") " Jul 29 21:25:10.773903 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.773873 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:25:10.774052 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.774024 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:25:10.774141 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.774068 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config" (OuterVolumeSpecName: "console-config") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:25:10.774141 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.774122 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca" (OuterVolumeSpecName: "service-ca") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:25:10.775968 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.775907 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c" (OuterVolumeSpecName: "kube-api-access-z4c2c") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "kube-api-access-z4c2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 21:25:10.775968 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.775949 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:25:10.776097 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.775967 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f0239014-0e17-49d6-92e3-0ff923388ae2" (UID: "f0239014-0e17-49d6-92e3-0ff923388ae2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 21:25:10.875113 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875087 2527 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-trusted-ca-bundle\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875113 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875113 2527 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-oauth-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875122 2527 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-console-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875131 2527 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-serving-cert\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875139 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4c2c\" (UniqueName: \"kubernetes.io/projected/f0239014-0e17-49d6-92e3-0ff923388ae2-kube-api-access-z4c2c\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875148 2527 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0239014-0e17-49d6-92e3-0ff923388ae2-console-oauth-config\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:10.875272 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:10.875156 2527 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0239014-0e17-49d6-92e3-0ff923388ae2-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:25:11.112276 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.112236 2527 generic.go:356] "Generic (PLEG): container finished" podID="f0239014-0e17-49d6-92e3-0ff923388ae2" containerID="a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6" exitCode=0 Jul 29 21:25:11.112449 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.112281 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79f69d98b9-rp82b" Jul 29 21:25:11.112449 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.112284 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f69d98b9-rp82b" event={"ID":"f0239014-0e17-49d6-92e3-0ff923388ae2","Type":"ContainerDied","Data":"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6"} Jul 29 21:25:11.112449 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.112342 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79f69d98b9-rp82b" event={"ID":"f0239014-0e17-49d6-92e3-0ff923388ae2","Type":"ContainerDied","Data":"42a58a2d28e2d69100ba6044761871740ceb068c0b66afe19279ddb1361cf55e"} Jul 29 21:25:11.112449 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.112369 2527 scope.go:117] "RemoveContainer" containerID="a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6" Jul 29 21:25:11.120617 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.120599 2527 scope.go:117] "RemoveContainer" containerID="a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6" Jul 29 21:25:11.120905 ip-10-0-141-233 kubenswrapper[2527]: E0729 21:25:11.120883 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6\": container with ID starting with a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6 not found: ID does not exist" containerID="a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6" Jul 29 21:25:11.121049 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.120917 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6"} err="failed to get container status \"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6\": rpc error: code = NotFound desc = could not find container \"a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6\": container with ID starting with a51ec8892eea40ca784ad903d90d4c9f6b51822bb79fd91270adc7c39ca260b6 not found: ID does not exist" Jul 29 21:25:11.129408 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.129387 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:25:11.135895 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:11.135875 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79f69d98b9-rp82b"] Jul 29 21:25:12.852856 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:25:12.852814 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0239014-0e17-49d6-92e3-0ff923388ae2" path="/var/lib/kubelet/pods/f0239014-0e17-49d6-92e3-0ff923388ae2/volumes" Jul 29 21:26:24.753300 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:26:24.753273 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:26:24.759537 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:26:24.759513 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:26:24.761975 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:26:24.761953 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:26:24.768286 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:26:24.768266 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:31:24.779265 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:31:24.779183 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:31:24.785571 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:31:24.785548 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:31:24.786265 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:31:24.786245 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:31:24.793339 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:31:24.793309 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:36:24.806431 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:36:24.806406 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:36:24.811302 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:36:24.811272 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:36:24.813240 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:36:24.813220 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:36:24.817908 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:36:24.817886 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:41:24.832658 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:41:24.832630 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:41:24.834907 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:41:24.834882 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:41:24.839517 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:41:24.839492 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:41:24.841832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:41:24.841814 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:46:24.861955 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:46:24.861906 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:46:24.864600 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:46:24.864581 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:46:24.868239 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:46:24.868212 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:46:24.870815 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:46:24.870797 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:49:47.686072 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.686039 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh"] Jul 29 21:49:47.686526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.686331 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0239014-0e17-49d6-92e3-0ff923388ae2" containerName="console" Jul 29 21:49:47.686526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.686342 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0239014-0e17-49d6-92e3-0ff923388ae2" containerName="console" Jul 29 21:49:47.686526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.686395 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0239014-0e17-49d6-92e3-0ff923388ae2" containerName="console" Jul 29 21:49:47.689165 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.689149 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:47.691775 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.691735 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:49:47.691889 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.691873 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:49:47.698285 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.698257 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh"] Jul 29 21:49:47.790373 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.790328 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-972150f2-2zzrh\" (UID: \"8cecb50d-d233-4913-9154-d273aa8c232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:47.891739 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.891697 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-972150f2-2zzrh\" (UID: \"8cecb50d-d233-4913-9154-d273aa8c232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:47.892437 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:47.892418 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-972150f2-2zzrh\" (UID: \"8cecb50d-d233-4913-9154-d273aa8c232f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:48.000600 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:48.000507 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:48.131307 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:48.131277 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh"] Jul 29 21:49:48.134589 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:49:48.134557 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cecb50d_d233_4913_9154_d273aa8c232f.slice/crio-6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631 WatchSource:0}: Error finding container 6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631: Status 404 returned error can't find the container with id 6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631 Jul 29 21:49:48.136490 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:48.136474 2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 21:49:48.296452 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:48.296368 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" event={"ID":"8cecb50d-d233-4913-9154-d273aa8c232f","Type":"ContainerStarted","Data":"6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631"} Jul 29 21:49:52.316284 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:52.316242 2527 generic.go:356] "Generic (PLEG): container finished" podID="8cecb50d-d233-4913-9154-d273aa8c232f" containerID="f52c9a3a9e3be514beb75747bb6d2c8d6c2bc589c65a6bef99344c15064cf9fa" exitCode=0 Jul 29 21:49:52.316679 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:52.316329 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" event={"ID":"8cecb50d-d233-4913-9154-d273aa8c232f","Type":"ContainerDied","Data":"f52c9a3a9e3be514beb75747bb6d2c8d6c2bc589c65a6bef99344c15064cf9fa"} Jul 29 21:49:53.441479 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:53.441457 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:53.537901 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:53.537866 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca\") pod \"8cecb50d-d233-4913-9154-d273aa8c232f\" (UID: \"8cecb50d-d233-4913-9154-d273aa8c232f\") " Jul 29 21:49:53.538266 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:53.538243 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cecb50d-d233-4913-9154-d273aa8c232f" (UID: "8cecb50d-d233-4913-9154-d273aa8c232f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:49:53.639124 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:53.639090 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cecb50d-d233-4913-9154-d273aa8c232f-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:49:54.326178 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:54.326146 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" event={"ID":"8cecb50d-d233-4913-9154-d273aa8c232f","Type":"ContainerDied","Data":"6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631"} Jul 29 21:49:54.326348 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:54.326185 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh" Jul 29 21:49:54.326348 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:49:54.326195 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bd87f964db03e5eb75ef58a1a22ff64f30a556a13d1238f49475346f0cac631" Jul 29 21:50:04.191019 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:04.190984 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh"] Jul 29 21:50:04.193951 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:04.193908 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-972150f2-2zzrh"] Jul 29 21:50:04.852736 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:04.852694 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cecb50d-d233-4913-9154-d273aa8c232f" path="/var/lib/kubelet/pods/8cecb50d-d233-4913-9154-d273aa8c232f/volumes" Jul 29 21:50:33.524562 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.524524 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z"] Jul 29 21:50:33.524939 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.524799 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cecb50d-d233-4913-9154-d273aa8c232f" containerName="revoke-keys" Jul 29 21:50:33.524939 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.524811 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cecb50d-d233-4913-9154-d273aa8c232f" containerName="revoke-keys" Jul 29 21:50:33.524939 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.524861 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cecb50d-d233-4913-9154-d273aa8c232f" containerName="revoke-keys" Jul 29 21:50:33.527872 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.527856 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:33.530128 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.530067 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:50:33.530269 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.530221 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:50:33.536652 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.536629 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z"] Jul 29 21:50:33.568695 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.568667 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z\" (UID: \"42a68045-19d4-4617-b553-4f0f2dde1d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:33.669331 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.669295 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z\" (UID: \"42a68045-19d4-4617-b553-4f0f2dde1d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:33.669888 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.669868 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z\" (UID: \"42a68045-19d4-4617-b553-4f0f2dde1d53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:33.839289 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.839194 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:33.972290 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:33.972262 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z"] Jul 29 21:50:33.974701 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:50:33.974664 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a68045_19d4_4617_b553_4f0f2dde1d53.slice/crio-fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621 WatchSource:0}: Error finding container fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621: Status 404 returned error can't find the container with id fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621 Jul 29 21:50:34.492150 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:34.492057 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" event={"ID":"42a68045-19d4-4617-b553-4f0f2dde1d53","Type":"ContainerStarted","Data":"6c6943e0575475265a6e2cb54d53e11443bab4db6d4aa0d9e76230c2c8ad978a"} Jul 29 21:50:34.492150 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:34.492112 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" event={"ID":"42a68045-19d4-4617-b553-4f0f2dde1d53","Type":"ContainerStarted","Data":"fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621"} Jul 29 21:50:34.509065 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:34.509016 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" podStartSLOduration=1.50899691 podStartE2EDuration="1.50899691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:50:34.508178558 +0000 UTC m=+2050.198610377" watchObservedRunningTime="2026-07-29 21:50:34.50899691 +0000 UTC m=+2050.199428728" Jul 29 21:50:35.498035 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:35.498000 2527 generic.go:356] "Generic (PLEG): container finished" podID="42a68045-19d4-4617-b553-4f0f2dde1d53" containerID="6c6943e0575475265a6e2cb54d53e11443bab4db6d4aa0d9e76230c2c8ad978a" exitCode=0 Jul 29 21:50:35.498515 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:35.498089 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" event={"ID":"42a68045-19d4-4617-b553-4f0f2dde1d53","Type":"ContainerDied","Data":"6c6943e0575475265a6e2cb54d53e11443bab4db6d4aa0d9e76230c2c8ad978a"} Jul 29 21:50:36.636141 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:36.636113 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:36.689006 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:36.688973 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca\") pod \"42a68045-19d4-4617-b553-4f0f2dde1d53\" (UID: \"42a68045-19d4-4617-b553-4f0f2dde1d53\") " Jul 29 21:50:36.689354 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:36.689330 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42a68045-19d4-4617-b553-4f0f2dde1d53" (UID: "42a68045-19d4-4617-b553-4f0f2dde1d53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:50:36.789526 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:36.789427 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a68045-19d4-4617-b553-4f0f2dde1d53-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:50:37.509215 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:37.509185 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" Jul 29 21:50:37.509425 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:37.509217 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z" event={"ID":"42a68045-19d4-4617-b553-4f0f2dde1d53","Type":"ContainerDied","Data":"fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621"} Jul 29 21:50:37.509425 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:37.509265 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbe8c593393232f5944a3d309f6b0ff4365b0a6beafc8561897310bbde21a621" Jul 29 21:50:40.410289 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:40.410250 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z"] Jul 29 21:50:40.414041 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:40.414014 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9394560f-p2k9z"] Jul 29 21:50:40.852172 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:50:40.852144 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a68045-19d4-4617-b553-4f0f2dde1d53" path="/var/lib/kubelet/pods/42a68045-19d4-4617-b553-4f0f2dde1d53/volumes" Jul 29 21:51:09.498853 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.498760 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt"] Jul 29 21:51:09.499450 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.499185 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a68045-19d4-4617-b553-4f0f2dde1d53" containerName="revoke-keys" Jul 29 21:51:09.499450 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.499205 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a68045-19d4-4617-b553-4f0f2dde1d53" containerName="revoke-keys" Jul 29 21:51:09.499450 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.499291 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a68045-19d4-4617-b553-4f0f2dde1d53" containerName="revoke-keys" Jul 29 21:51:09.502261 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.502240 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:09.505076 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.505056 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:51:09.505378 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.505360 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:51:09.509845 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.509817 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt\" (UID: \"45a2cc05-2fb9-40fa-8f8f-a02e0511a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:09.511872 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.511852 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt"] Jul 29 21:51:09.611109 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.611075 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt\" (UID: \"45a2cc05-2fb9-40fa-8f8f-a02e0511a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:09.611649 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.611628 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt\" (UID: \"45a2cc05-2fb9-40fa-8f8f-a02e0511a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:09.812814 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.812718 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:09.946746 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:09.946720 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt"] Jul 29 21:51:09.949267 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:51:09.949221 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45a2cc05_2fb9_40fa_8f8f_a02e0511a213.slice/crio-e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447 WatchSource:0}: Error finding container e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447: Status 404 returned error can't find the container with id e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447 Jul 29 21:51:10.651511 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:10.651477 2527 generic.go:356] "Generic (PLEG): container finished" podID="45a2cc05-2fb9-40fa-8f8f-a02e0511a213" containerID="32a18a12af6caf5c851f83be2d2934a53403e0999cbff10d5850c95e1e6439ad" exitCode=0 Jul 29 21:51:10.651889 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:10.651564 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" event={"ID":"45a2cc05-2fb9-40fa-8f8f-a02e0511a213","Type":"ContainerDied","Data":"32a18a12af6caf5c851f83be2d2934a53403e0999cbff10d5850c95e1e6439ad"} Jul 29 21:51:10.651889 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:10.651622 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" event={"ID":"45a2cc05-2fb9-40fa-8f8f-a02e0511a213","Type":"ContainerStarted","Data":"e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447"} Jul 29 21:51:11.781631 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:11.781609 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:11.830913 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:11.830881 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca\") pod \"45a2cc05-2fb9-40fa-8f8f-a02e0511a213\" (UID: \"45a2cc05-2fb9-40fa-8f8f-a02e0511a213\") " Jul 29 21:51:11.831257 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:11.831232 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "45a2cc05-2fb9-40fa-8f8f-a02e0511a213" (UID: "45a2cc05-2fb9-40fa-8f8f-a02e0511a213"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:51:11.931832 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:11.931741 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/45a2cc05-2fb9-40fa-8f8f-a02e0511a213-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:51:12.661545 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:12.661514 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" Jul 29 21:51:12.661545 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:12.661533 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt" event={"ID":"45a2cc05-2fb9-40fa-8f8f-a02e0511a213","Type":"ContainerDied","Data":"e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447"} Jul 29 21:51:12.661759 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:12.661576 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e10953eb32922c713ec9a17f581cb26c0747cb2cb031cd12565fe11d83632447" Jul 29 21:51:15.559094 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:15.559062 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt"] Jul 29 21:51:15.561538 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:15.561514 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0856dcaa-p5hnt"] Jul 29 21:51:16.852392 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:16.852360 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45a2cc05-2fb9-40fa-8f8f-a02e0511a213" path="/var/lib/kubelet/pods/45a2cc05-2fb9-40fa-8f8f-a02e0511a213/volumes" Jul 29 21:51:24.888517 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:24.888487 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:51:24.888960 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:24.888611 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:51:24.895202 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:24.895181 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:51:24.895319 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:24.895253 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:51:44.657773 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.657729 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66"] Jul 29 21:51:44.658303 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.658040 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45a2cc05-2fb9-40fa-8f8f-a02e0511a213" containerName="revoke-keys" Jul 29 21:51:44.658303 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.658051 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="45a2cc05-2fb9-40fa-8f8f-a02e0511a213" containerName="revoke-keys" Jul 29 21:51:44.658303 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.658110 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="45a2cc05-2fb9-40fa-8f8f-a02e0511a213" containerName="revoke-keys" Jul 29 21:51:44.661100 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.661084 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:44.663292 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.663272 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:51:44.663418 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.663333 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:51:44.668882 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.668859 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66"] Jul 29 21:51:44.798755 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.798720 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-48a37a48-gpv66\" (UID: \"eeb4b856-c725-4188-80ac-32bfd4c9a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:44.899960 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.899901 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-48a37a48-gpv66\" (UID: \"eeb4b856-c725-4188-80ac-32bfd4c9a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:44.900507 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.900485 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-48a37a48-gpv66\" (UID: \"eeb4b856-c725-4188-80ac-32bfd4c9a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:44.971612 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:44.971523 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:45.102409 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:45.102262 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66"] Jul 29 21:51:45.105261 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:51:45.105235 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb4b856_c725_4188_80ac_32bfd4c9a7fe.slice/crio-db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30 WatchSource:0}: Error finding container db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30: Status 404 returned error can't find the container with id db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30 Jul 29 21:51:45.808255 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:45.808217 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" event={"ID":"eeb4b856-c725-4188-80ac-32bfd4c9a7fe","Type":"ContainerStarted","Data":"cecf60d945f862f682ac09813062b3211d230512024ea371838de592ab234e0c"} Jul 29 21:51:45.808640 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:45.808275 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" event={"ID":"eeb4b856-c725-4188-80ac-32bfd4c9a7fe","Type":"ContainerStarted","Data":"db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30"} Jul 29 21:51:45.825160 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:45.825113 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" podStartSLOduration=1.825096045 podStartE2EDuration="1.825096045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:51:45.823256385 +0000 UTC m=+2121.513688203" watchObservedRunningTime="2026-07-29 21:51:45.825096045 +0000 UTC m=+2121.515527863" Jul 29 21:51:46.814706 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:46.814674 2527 generic.go:356] "Generic (PLEG): container finished" podID="eeb4b856-c725-4188-80ac-32bfd4c9a7fe" containerID="cecf60d945f862f682ac09813062b3211d230512024ea371838de592ab234e0c" exitCode=0 Jul 29 21:51:46.815266 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:46.814765 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" event={"ID":"eeb4b856-c725-4188-80ac-32bfd4c9a7fe","Type":"ContainerDied","Data":"cecf60d945f862f682ac09813062b3211d230512024ea371838de592ab234e0c"} Jul 29 21:51:47.943620 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:47.943595 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:48.025830 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.025798 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca\") pod \"eeb4b856-c725-4188-80ac-32bfd4c9a7fe\" (UID: \"eeb4b856-c725-4188-80ac-32bfd4c9a7fe\") " Jul 29 21:51:48.026210 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.026189 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eeb4b856-c725-4188-80ac-32bfd4c9a7fe" (UID: "eeb4b856-c725-4188-80ac-32bfd4c9a7fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:51:48.127012 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.126978 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eeb4b856-c725-4188-80ac-32bfd4c9a7fe-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:51:48.825101 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.825070 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" event={"ID":"eeb4b856-c725-4188-80ac-32bfd4c9a7fe","Type":"ContainerDied","Data":"db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30"} Jul 29 21:51:48.825101 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.825094 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66" Jul 29 21:51:48.825317 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:48.825120 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db528dc77e966a4d156f25ca814411401a98300c9ada9c9acea13eee68171f30" Jul 29 21:51:51.682851 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:51.682820 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66"] Jul 29 21:51:51.684160 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:51.684140 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-48a37a48-gpv66"] Jul 29 21:51:52.851752 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:51:52.851713 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeb4b856-c725-4188-80ac-32bfd4c9a7fe" path="/var/lib/kubelet/pods/eeb4b856-c725-4188-80ac-32bfd4c9a7fe/volumes" Jul 29 21:54:37.834364 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.834324 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4"] Jul 29 21:54:37.834778 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.834592 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeb4b856-c725-4188-80ac-32bfd4c9a7fe" containerName="revoke-keys" Jul 29 21:54:37.834778 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.834602 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeb4b856-c725-4188-80ac-32bfd4c9a7fe" containerName="revoke-keys" Jul 29 21:54:37.834778 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.834659 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeb4b856-c725-4188-80ac-32bfd4c9a7fe" containerName="revoke-keys" Jul 29 21:54:37.837416 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.837399 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:37.841125 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.841104 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:54:37.844733 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.844716 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:54:37.853915 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.853885 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4"] Jul 29 21:54:37.878659 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.878616 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4\" (UID: \"9a547fc4-0682-4564-8423-05703a71d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:37.980055 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.980013 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4\" (UID: \"9a547fc4-0682-4564-8423-05703a71d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:37.980610 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:37.980589 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4\" (UID: \"9a547fc4-0682-4564-8423-05703a71d627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:38.147554 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:38.147517 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:38.282585 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:38.282549 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4"] Jul 29 21:54:38.284784 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:54:38.284754 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a547fc4_0682_4564_8423_05703a71d627.slice/crio-58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00 WatchSource:0}: Error finding container 58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00: Status 404 returned error can't find the container with id 58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00 Jul 29 21:54:38.570648 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:38.570612 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" event={"ID":"9a547fc4-0682-4564-8423-05703a71d627","Type":"ContainerStarted","Data":"04ccdd1c867cc90d62688836549fbf3a36ec3cc6c385b9c363f42dc1b4ef4202"} Jul 29 21:54:38.570843 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:38.570676 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" event={"ID":"9a547fc4-0682-4564-8423-05703a71d627","Type":"ContainerStarted","Data":"58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00"} Jul 29 21:54:38.587723 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:38.587677 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" podStartSLOduration=1.587657529 podStartE2EDuration="1.587657529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:54:38.586445238 +0000 UTC m=+2294.276877057" watchObservedRunningTime="2026-07-29 21:54:38.587657529 +0000 UTC m=+2294.278089348" Jul 29 21:54:39.576038 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:39.575999 2527 generic.go:356] "Generic (PLEG): container finished" podID="9a547fc4-0682-4564-8423-05703a71d627" containerID="04ccdd1c867cc90d62688836549fbf3a36ec3cc6c385b9c363f42dc1b4ef4202" exitCode=0 Jul 29 21:54:39.576416 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:39.576045 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" event={"ID":"9a547fc4-0682-4564-8423-05703a71d627","Type":"ContainerDied","Data":"04ccdd1c867cc90d62688836549fbf3a36ec3cc6c385b9c363f42dc1b4ef4202"} Jul 29 21:54:40.707315 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:40.707290 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:40.800752 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:40.800718 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca\") pod \"9a547fc4-0682-4564-8423-05703a71d627\" (UID: \"9a547fc4-0682-4564-8423-05703a71d627\") " Jul 29 21:54:40.801157 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:40.801130 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a547fc4-0682-4564-8423-05703a71d627" (UID: "9a547fc4-0682-4564-8423-05703a71d627"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:54:40.901900 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:40.901867 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a547fc4-0682-4564-8423-05703a71d627-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:54:41.586092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:41.586046 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" Jul 29 21:54:41.586092 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:41.586066 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4" event={"ID":"9a547fc4-0682-4564-8423-05703a71d627","Type":"ContainerDied","Data":"58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00"} Jul 29 21:54:41.586330 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:41.586121 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b39951b47178507cc3de244e71a2d5b7c877d94118d68f9491d1468c870b00" Jul 29 21:54:48.628583 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:48.628546 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4"] Jul 29 21:54:48.631843 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:48.631814 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d5dee3c8-5pjw4"] Jul 29 21:54:48.852874 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:54:48.852838 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a547fc4-0682-4564-8423-05703a71d627" path="/var/lib/kubelet/pods/9a547fc4-0682-4564-8423-05703a71d627/volumes" Jul 29 21:56:24.915497 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:56:24.915470 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:56:24.916191 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:56:24.916175 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 21:56:24.921946 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:56:24.921899 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:56:24.922433 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:56:24.922412 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 21:56:24.972333 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:56:24.972310 2527 scope.go:117] "RemoveContainer" containerID="f52c9a3a9e3be514beb75747bb6d2c8d6c2bc589c65a6bef99344c15064cf9fa" Jul 29 21:57:24.991084 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:57:24.991051 2527 scope.go:117] "RemoveContainer" containerID="32a18a12af6caf5c851f83be2d2934a53403e0999cbff10d5850c95e1e6439ad" Jul 29 21:57:24.998949 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:57:24.998915 2527 scope.go:117] "RemoveContainer" containerID="6c6943e0575475265a6e2cb54d53e11443bab4db6d4aa0d9e76230c2c8ad978a" Jul 29 21:58:08.517497 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.517453 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc"] Jul 29 21:58:08.517998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.517750 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a547fc4-0682-4564-8423-05703a71d627" containerName="revoke-keys" Jul 29 21:58:08.517998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.517762 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a547fc4-0682-4564-8423-05703a71d627" containerName="revoke-keys" Jul 29 21:58:08.517998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.517810 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a547fc4-0682-4564-8423-05703a71d627" containerName="revoke-keys" Jul 29 21:58:08.520648 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.520633 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:08.522998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.522976 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 21:58:08.522998 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.522985 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 21:58:08.528395 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.528361 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc"] Jul 29 21:58:08.598776 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.598744 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc\" (UID: \"75ee97f1-85a7-4de2-8109-b8747f436302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:08.699342 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.699297 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc\" (UID: \"75ee97f1-85a7-4de2-8109-b8747f436302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:08.699967 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.699909 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc\" (UID: \"75ee97f1-85a7-4de2-8109-b8747f436302\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:08.831249 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.831155 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:08.966060 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:08.966027 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc"] Jul 29 21:58:08.969492 ip-10-0-141-233 kubenswrapper[2527]: W0729 21:58:08.969463 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ee97f1_85a7_4de2_8109_b8747f436302.slice/crio-645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade WatchSource:0}: Error finding container 645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade: Status 404 returned error can't find the container with id 645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade Jul 29 21:58:09.498323 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:09.498236 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" event={"ID":"75ee97f1-85a7-4de2-8109-b8747f436302","Type":"ContainerStarted","Data":"6e4a79bec072e4f7c21d54ff1bd63d365f8086c28b5767bccc52a56e9e284c5c"} Jul 29 21:58:09.498323 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:09.498292 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" event={"ID":"75ee97f1-85a7-4de2-8109-b8747f436302","Type":"ContainerStarted","Data":"645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade"} Jul 29 21:58:09.514746 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:09.514679 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" podStartSLOduration=1.514662077 podStartE2EDuration="1.514662077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 21:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 21:58:09.514408117 +0000 UTC m=+2505.204839934" watchObservedRunningTime="2026-07-29 21:58:09.514662077 +0000 UTC m=+2505.205093895" Jul 29 21:58:10.504115 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:10.504082 2527 generic.go:356] "Generic (PLEG): container finished" podID="75ee97f1-85a7-4de2-8109-b8747f436302" containerID="6e4a79bec072e4f7c21d54ff1bd63d365f8086c28b5767bccc52a56e9e284c5c" exitCode=0 Jul 29 21:58:10.504533 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:10.504159 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" event={"ID":"75ee97f1-85a7-4de2-8109-b8747f436302","Type":"ContainerDied","Data":"6e4a79bec072e4f7c21d54ff1bd63d365f8086c28b5767bccc52a56e9e284c5c"} Jul 29 21:58:11.637875 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:11.637848 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:11.725479 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:11.725444 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca\") pod \"75ee97f1-85a7-4de2-8109-b8747f436302\" (UID: \"75ee97f1-85a7-4de2-8109-b8747f436302\") " Jul 29 21:58:11.725825 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:11.725801 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75ee97f1-85a7-4de2-8109-b8747f436302" (UID: "75ee97f1-85a7-4de2-8109-b8747f436302"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 21:58:11.826314 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:11.826225 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75ee97f1-85a7-4de2-8109-b8747f436302-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 21:58:12.514767 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:12.514734 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" Jul 29 21:58:12.514980 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:12.514738 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc" event={"ID":"75ee97f1-85a7-4de2-8109-b8747f436302","Type":"ContainerDied","Data":"645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade"} Jul 29 21:58:12.514980 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:12.514859 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="645765ad79c68199dd16a225976632d1f71e784ec8ad433b850c34ba3989fade" Jul 29 21:58:24.413244 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:24.413115 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc"] Jul 29 21:58:24.416346 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:24.416320 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9bee5593-nj7pc"] Jul 29 21:58:24.851790 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:24.851762 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75ee97f1-85a7-4de2-8109-b8747f436302" path="/var/lib/kubelet/pods/75ee97f1-85a7-4de2-8109-b8747f436302/volumes" Jul 29 21:58:25.019656 ip-10-0-141-233 kubenswrapper[2527]: I0729 21:58:25.019617 2527 scope.go:117] "RemoveContainer" containerID="cecf60d945f862f682ac09813062b3211d230512024ea371838de592ab234e0c" Jul 29 22:01:24.939572 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:01:24.939495 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:01:24.941774 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:01:24.941752 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:01:24.946331 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:01:24.946307 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:01:24.948554 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:01:24.948535 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:01:25.052074 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:01:25.052040 2527 scope.go:117] "RemoveContainer" containerID="04ccdd1c867cc90d62688836549fbf3a36ec3cc6c385b9c363f42dc1b4ef4202" Jul 29 22:02:07.033266 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.033233 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:02:07.033740 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.033509 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75ee97f1-85a7-4de2-8109-b8747f436302" containerName="revoke-keys" Jul 29 22:02:07.033740 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.033519 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="75ee97f1-85a7-4de2-8109-b8747f436302" containerName="revoke-keys" Jul 29 22:02:07.033740 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.033566 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="75ee97f1-85a7-4de2-8109-b8747f436302" containerName="revoke-keys" Jul 29 22:02:07.036462 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.036441 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.038908 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.038884 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c24a76\"/\"default-dockercfg-l2jgr\"" Jul 29 22:02:07.039584 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.039564 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c24a76\"/\"kube-root-ca.crt\"" Jul 29 22:02:07.039667 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.039590 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c24a76\"/\"openshift-service-ca.crt\"" Jul 29 22:02:07.039667 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.039647 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c24a76\"/\"test-model-c24a76-kserve-self-signed-certs\"" Jul 29 22:02:07.049874 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.049846 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:02:07.138000 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.137963 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.138000 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.138006 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.138221 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.138036 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.138221 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.138109 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.138221 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.138134 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.138221 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.138179 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppcfs\" (UniqueName: \"kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239164 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239107 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239164 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239182 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239420 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239197 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239420 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239227 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppcfs\" (UniqueName: \"kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239420 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239251 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239420 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239272 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239634 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239576 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239634 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239590 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.239731 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.239668 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.241628 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.241608 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.241943 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.241905 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.249631 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.249608 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppcfs\" (UniqueName: \"kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs\") pod \"test-model-c24a76-kserve-5c98f97dc9-fwjwj\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.345443 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.345334 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:07.482342 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.482304 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:02:07.485423 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:02:07.485390 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a2fb14b_f54c_4ed2_acda_84f90aaa0134.slice/crio-be5125302c726bbaa1e0dc5c617d1f0a673ccaca8337fef3b9f32e3cd1b6bf9b WatchSource:0}: Error finding container be5125302c726bbaa1e0dc5c617d1f0a673ccaca8337fef3b9f32e3cd1b6bf9b: Status 404 returned error can't find the container with id be5125302c726bbaa1e0dc5c617d1f0a673ccaca8337fef3b9f32e3cd1b6bf9b Jul 29 22:02:07.487258 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.487239 2527 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 22:02:07.513237 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:07.513205 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" event={"ID":"1a2fb14b-f54c-4ed2-acda-84f90aaa0134","Type":"ContainerStarted","Data":"be5125302c726bbaa1e0dc5c617d1f0a673ccaca8337fef3b9f32e3cd1b6bf9b"} Jul 29 22:02:09.529599 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:09.529564 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" event={"ID":"1a2fb14b-f54c-4ed2-acda-84f90aaa0134","Type":"ContainerStarted","Data":"d4c9cd8b5fdbb5cce4edc01e2b79ba56e7a6ecfb940c4472d5fcad4a8c6356bf"} Jul 29 22:02:09.549108 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:09.549055 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" podStartSLOduration=1.466359813 podStartE2EDuration="2.549039275s" totalImagesPullingTime="1.082679462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:02:09.547419469 +0000 UTC m=+2745.237851280" watchObservedRunningTime="2026-07-29 22:02:09.549039275 +0000 UTC m=+2745.239471093" Jul 29 22:02:17.346502 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:17.346465 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:17.347024 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:17.346523 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:17.358733 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:17.358704 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:17.575266 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:17.575231 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:02:53.632141 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:53.632060 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:02:53.775455 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:53.775421 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw"] Jul 29 22:02:53.779405 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:53.779381 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:53.782037 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:02:53.782001 2527 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw\" is forbidden: User \"system:node:ip-10-0-141-233.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-233.ec2.internal' and this object" podUID="358a61d8-6464-4f5f-bc83-0fc61e76c766" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:53.782154 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:02:53.782067 2527 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-cleanup-dockercfg-hxbps\" is forbidden: User \"system:node:ip-10-0-141-233.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-233.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" type="*v1.Secret" Jul 29 22:02:53.782154 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:02:53.782094 2527 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-141-233.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-233.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 29 22:02:53.804570 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:53.804540 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw"] Jul 29 22:02:53.942216 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:53.942125 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw\" (UID: \"358a61d8-6464-4f5f-bc83-0fc61e76c766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:54.043543 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:54.043483 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw\" (UID: \"358a61d8-6464-4f5f-bc83-0fc61e76c766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:54.621992 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:54.621962 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 22:02:54.801636 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:54.801603 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 22:02:54.805069 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:54.805045 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw\" (UID: \"358a61d8-6464-4f5f-bc83-0fc61e76c766\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:54.988703 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:54.988618 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:55.125109 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:55.125084 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw"] Jul 29 22:02:55.127766 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:02:55.127737 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358a61d8_6464_4f5f_bc83_0fc61e76c766.slice/crio-74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58 WatchSource:0}: Error finding container 74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58: Status 404 returned error can't find the container with id 74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58 Jul 29 22:02:55.731326 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:55.731284 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" event={"ID":"358a61d8-6464-4f5f-bc83-0fc61e76c766","Type":"ContainerStarted","Data":"ed6bfb4b3b16a44adcc34410d1d21ad999030035e580e5e877ea6c7ac36092f4"} Jul 29 22:02:55.731525 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:55.731354 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" event={"ID":"358a61d8-6464-4f5f-bc83-0fc61e76c766","Type":"ContainerStarted","Data":"74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58"} Jul 29 22:02:55.750106 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:55.750047 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" podStartSLOduration=2.750028772 podStartE2EDuration="2.750028772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:02:55.748332035 +0000 UTC m=+2791.438763857" watchObservedRunningTime="2026-07-29 22:02:55.750028772 +0000 UTC m=+2791.440460589" Jul 29 22:02:56.736882 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:56.736848 2527 generic.go:356] "Generic (PLEG): container finished" podID="358a61d8-6464-4f5f-bc83-0fc61e76c766" containerID="ed6bfb4b3b16a44adcc34410d1d21ad999030035e580e5e877ea6c7ac36092f4" exitCode=0 Jul 29 22:02:56.737284 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:56.736922 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" event={"ID":"358a61d8-6464-4f5f-bc83-0fc61e76c766","Type":"ContainerDied","Data":"ed6bfb4b3b16a44adcc34410d1d21ad999030035e580e5e877ea6c7ac36092f4"} Jul 29 22:02:57.870011 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:57.869981 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:57.870426 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:57.870406 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca\") pod \"358a61d8-6464-4f5f-bc83-0fc61e76c766\" (UID: \"358a61d8-6464-4f5f-bc83-0fc61e76c766\") " Jul 29 22:02:57.870747 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:57.870723 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "358a61d8-6464-4f5f-bc83-0fc61e76c766" (UID: "358a61d8-6464-4f5f-bc83-0fc61e76c766"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:02:57.971194 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:57.971155 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/358a61d8-6464-4f5f-bc83-0fc61e76c766-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:02:58.746745 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:58.746715 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" Jul 29 22:02:58.746745 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:58.746731 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw" event={"ID":"358a61d8-6464-4f5f-bc83-0fc61e76c766","Type":"ContainerDied","Data":"74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58"} Jul 29 22:02:58.747010 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:02:58.746807 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74027fe6c042e29e9fad82cc6cdafdbdba417bb20d5fa93e9017b98f9d6daf58" Jul 29 22:03:04.616442 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:04.616402 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw"] Jul 29 22:03:04.618900 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:04.618874 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-25c9e8-srjzw"] Jul 29 22:03:04.852703 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:04.852660 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358a61d8-6464-4f5f-bc83-0fc61e76c766" path="/var/lib/kubelet/pods/358a61d8-6464-4f5f-bc83-0fc61e76c766/volumes" Jul 29 22:03:08.651088 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.651024 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" podUID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" containerName="main" containerID="cri-o://d4c9cd8b5fdbb5cce4edc01e2b79ba56e7a6ecfb940c4472d5fcad4a8c6356bf" gracePeriod=45 Jul 29 22:03:08.790358 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.790321 2527 generic.go:356] "Generic (PLEG): container finished" podID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" containerID="d4c9cd8b5fdbb5cce4edc01e2b79ba56e7a6ecfb940c4472d5fcad4a8c6356bf" exitCode=0 Jul 29 22:03:08.790512 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.790388 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" event={"ID":"1a2fb14b-f54c-4ed2-acda-84f90aaa0134","Type":"ContainerDied","Data":"d4c9cd8b5fdbb5cce4edc01e2b79ba56e7a6ecfb940c4472d5fcad4a8c6356bf"} Jul 29 22:03:08.900868 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.900844 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:03:08.948737 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948654 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.948737 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948697 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.948737 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948727 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.949036 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948782 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.949036 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948811 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.949036 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.948856 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppcfs\" (UniqueName: \"kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs\") pod \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\" (UID: \"1a2fb14b-f54c-4ed2-acda-84f90aaa0134\") " Jul 29 22:03:08.949036 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.949002 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home" (OuterVolumeSpecName: "home") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:03:08.949234 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.949054 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache" (OuterVolumeSpecName: "model-cache") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:03:08.949234 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.949129 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:03:08.949234 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.949158 2527 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-home\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:08.951229 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.951207 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 22:03:08.951229 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.951212 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm" (OuterVolumeSpecName: "dshm") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:03:08.951330 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:08.951227 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs" (OuterVolumeSpecName: "kube-api-access-ppcfs") pod "1a2fb14b-f54c-4ed2-acda-84f90aaa0134" (UID: "1a2fb14b-f54c-4ed2-acda-84f90aaa0134"). InnerVolumeSpecName "kube-api-access-ppcfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:03:09.049871 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.049832 2527 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-model-cache\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.049871 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.049863 2527 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tmp-dir\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.049871 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.049875 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppcfs\" (UniqueName: \"kubernetes.io/projected/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-kube-api-access-ppcfs\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.050138 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.049886 2527 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-tls-certs\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.050138 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.049897 2527 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a2fb14b-f54c-4ed2-acda-84f90aaa0134-dshm\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:09.796666 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.796638 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" Jul 29 22:03:09.797158 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.796679 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj" event={"ID":"1a2fb14b-f54c-4ed2-acda-84f90aaa0134","Type":"ContainerDied","Data":"be5125302c726bbaa1e0dc5c617d1f0a673ccaca8337fef3b9f32e3cd1b6bf9b"} Jul 29 22:03:09.797158 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.796737 2527 scope.go:117] "RemoveContainer" containerID="d4c9cd8b5fdbb5cce4edc01e2b79ba56e7a6ecfb940c4472d5fcad4a8c6356bf" Jul 29 22:03:09.819412 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.819383 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:03:09.824739 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:09.824713 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c24a76/test-model-c24a76-kserve-5c98f97dc9-fwjwj"] Jul 29 22:03:10.852617 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:10.852584 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" path="/var/lib/kubelet/pods/1a2fb14b-f54c-4ed2-acda-84f90aaa0134/volumes" Jul 29 22:03:12.219098 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219066 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv"] Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219330 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" containerName="main" Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219341 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" containerName="main" Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219362 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358a61d8-6464-4f5f-bc83-0fc61e76c766" containerName="revoke-keys" Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219368 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="358a61d8-6464-4f5f-bc83-0fc61e76c766" containerName="revoke-keys" Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219415 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="358a61d8-6464-4f5f-bc83-0fc61e76c766" containerName="revoke-keys" Jul 29 22:03:12.219468 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.219423 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a2fb14b-f54c-4ed2-acda-84f90aaa0134" containerName="main" Jul 29 22:03:12.223817 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.223798 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:12.226208 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.226188 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 22:03:12.226329 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.226313 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 22:03:12.234292 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.234262 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv"] Jul 29 22:03:12.275593 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.275561 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv\" (UID: \"64a0a1c9-9e6e-4f3c-adcd-2314201782b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:12.376586 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.376547 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv\" (UID: \"64a0a1c9-9e6e-4f3c-adcd-2314201782b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:12.377193 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.377173 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv\" (UID: \"64a0a1c9-9e6e-4f3c-adcd-2314201782b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:12.534315 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.534227 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:12.665442 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.665414 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv"] Jul 29 22:03:12.668027 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:03:12.667993 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a0a1c9_9e6e_4f3c_adcd_2314201782b0.slice/crio-dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d WatchSource:0}: Error finding container dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d: Status 404 returned error can't find the container with id dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d Jul 29 22:03:12.811359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:12.811275 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" event={"ID":"64a0a1c9-9e6e-4f3c-adcd-2314201782b0","Type":"ContainerStarted","Data":"dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d"} Jul 29 22:03:13.817009 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:13.816979 2527 generic.go:356] "Generic (PLEG): container finished" podID="64a0a1c9-9e6e-4f3c-adcd-2314201782b0" containerID="27a66719ca79afbb29df114288c7df54de6f637cec54b41cd0b9e5cd678d874f" exitCode=0 Jul 29 22:03:13.817403 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:13.817068 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" event={"ID":"64a0a1c9-9e6e-4f3c-adcd-2314201782b0","Type":"ContainerDied","Data":"27a66719ca79afbb29df114288c7df54de6f637cec54b41cd0b9e5cd678d874f"} Jul 29 22:03:14.953507 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:14.953476 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:14.999819 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:14.999792 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca\") pod \"64a0a1c9-9e6e-4f3c-adcd-2314201782b0\" (UID: \"64a0a1c9-9e6e-4f3c-adcd-2314201782b0\") " Jul 29 22:03:15.000146 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:15.000125 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "64a0a1c9-9e6e-4f3c-adcd-2314201782b0" (UID: "64a0a1c9-9e6e-4f3c-adcd-2314201782b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:03:15.101212 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:15.101183 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a0a1c9-9e6e-4f3c-adcd-2314201782b0-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:03:15.826393 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:15.826362 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" Jul 29 22:03:15.826393 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:15.826390 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv" event={"ID":"64a0a1c9-9e6e-4f3c-adcd-2314201782b0","Type":"ContainerDied","Data":"dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d"} Jul 29 22:03:15.826650 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:15.826436 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc35536d60aee3e86d600081856ccd3aa850dc25b27580c2ff9bea96f78cea9d" Jul 29 22:03:23.173949 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:23.173364 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv"] Jul 29 22:03:23.173949 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:23.173427 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c24a76-xjgjv"] Jul 29 22:03:24.852098 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:03:24.852069 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a0a1c9-9e6e-4f3c-adcd-2314201782b0" path="/var/lib/kubelet/pods/64a0a1c9-9e6e-4f3c-adcd-2314201782b0/volumes" Jul 29 22:04:10.646759 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.646725 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:04:10.647251 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.647024 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a0a1c9-9e6e-4f3c-adcd-2314201782b0" containerName="revoke-keys" Jul 29 22:04:10.647251 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.647036 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a0a1c9-9e6e-4f3c-adcd-2314201782b0" containerName="revoke-keys" Jul 29 22:04:10.647251 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.647084 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a0a1c9-9e6e-4f3c-adcd-2314201782b0" containerName="revoke-keys" Jul 29 22:04:10.649940 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.649914 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.653697 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.653675 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6ab119-dockercfg-gsjw9\"" Jul 29 22:04:10.653807 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.653718 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6ab119\"" Jul 29 22:04:10.664676 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.664647 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:04:10.758805 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.758776 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.758977 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.758813 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qjdr\" (UniqueName: \"kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.860174 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.860137 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.860174 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.860177 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qjdr\" (UniqueName: \"kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.860862 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.860835 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.870145 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.870113 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qjdr\" (UniqueName: \"kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr\") pod \"payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:10.959551 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:10.959441 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:11.090181 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:11.090137 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:04:11.093004 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:04:11.092978 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e7c7c3_6f8e_4d8c_b07b_971153f6639c.slice/crio-7fd8136efa2121aa907946a8282b274e05b43b050a76deb9a00a8a710758a9ad WatchSource:0}: Error finding container 7fd8136efa2121aa907946a8282b274e05b43b050a76deb9a00a8a710758a9ad: Status 404 returned error can't find the container with id 7fd8136efa2121aa907946a8282b274e05b43b050a76deb9a00a8a710758a9ad Jul 29 22:04:12.069827 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:12.069786 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" event={"ID":"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c","Type":"ContainerStarted","Data":"7fd8136efa2121aa907946a8282b274e05b43b050a76deb9a00a8a710758a9ad"} Jul 29 22:04:13.075402 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:13.075309 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" event={"ID":"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c","Type":"ContainerStarted","Data":"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6"} Jul 29 22:04:13.075753 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:13.075400 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:13.098485 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:13.098436 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" podStartSLOduration=1.5103977629999998 podStartE2EDuration="3.098418553s" totalImagesPullingTime="1.58802079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:04:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:04:13.095614017 +0000 UTC m=+2868.786045826" watchObservedRunningTime="2026-07-29 22:04:13.098418553 +0000 UTC m=+2868.788850371" Jul 29 22:04:24.081518 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:24.081441 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:04:25.088413 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:25.088381 2527 scope.go:117] "RemoveContainer" containerID="6e4a79bec072e4f7c21d54ff1bd63d365f8086c28b5767bccc52a56e9e284c5c" Jul 29 22:04:36.733852 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.733818 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr"] Jul 29 22:04:36.737196 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.737171 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:36.741218 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.741194 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 22:04:36.741348 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.741194 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 22:04:36.747111 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.747085 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr"] Jul 29 22:04:36.886289 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.886233 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr\" (UID: \"16e20c02-8287-4fd3-b078-0bc214c598f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:36.987054 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.986952 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr\" (UID: \"16e20c02-8287-4fd3-b078-0bc214c598f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:36.987770 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:36.987736 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr\" (UID: \"16e20c02-8287-4fd3-b078-0bc214c598f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:37.048044 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:37.048008 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:37.201712 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:37.201678 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr"] Jul 29 22:04:37.204129 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:04:37.204100 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e20c02_8287_4fd3_b078_0bc214c598f5.slice/crio-8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e WatchSource:0}: Error finding container 8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e: Status 404 returned error can't find the container with id 8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e Jul 29 22:04:38.185812 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:38.185776 2527 generic.go:356] "Generic (PLEG): container finished" podID="16e20c02-8287-4fd3-b078-0bc214c598f5" containerID="aebf8f5d27ddee14988a3256326e030b9a89a614c11d9c97cd82b5a1b7cb9880" exitCode=0 Jul 29 22:04:38.186239 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:38.185826 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" event={"ID":"16e20c02-8287-4fd3-b078-0bc214c598f5","Type":"ContainerDied","Data":"aebf8f5d27ddee14988a3256326e030b9a89a614c11d9c97cd82b5a1b7cb9880"} Jul 29 22:04:38.186239 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:38.185877 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" event={"ID":"16e20c02-8287-4fd3-b078-0bc214c598f5","Type":"ContainerStarted","Data":"8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e"} Jul 29 22:04:39.317245 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:39.317220 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:39.405663 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:39.405625 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca\") pod \"16e20c02-8287-4fd3-b078-0bc214c598f5\" (UID: \"16e20c02-8287-4fd3-b078-0bc214c598f5\") " Jul 29 22:04:39.406050 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:39.406028 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16e20c02-8287-4fd3-b078-0bc214c598f5" (UID: "16e20c02-8287-4fd3-b078-0bc214c598f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:04:39.506396 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:39.506301 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e20c02-8287-4fd3-b078-0bc214c598f5-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:04:40.194990 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:40.194958 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" Jul 29 22:04:40.194990 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:40.194968 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr" event={"ID":"16e20c02-8287-4fd3-b078-0bc214c598f5","Type":"ContainerDied","Data":"8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e"} Jul 29 22:04:40.195194 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:40.195018 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e9149bbe908f756148f0807a6382c3dae01628048e999a9255192f4460b7d8e" Jul 29 22:04:52.688531 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:52.688496 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr"] Jul 29 22:04:52.693752 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:52.693727 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c8e5f0-7mwsr"] Jul 29 22:04:52.852685 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:04:52.852649 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e20c02-8287-4fd3-b078-0bc214c598f5" path="/var/lib/kubelet/pods/16e20c02-8287-4fd3-b078-0bc214c598f5/volumes" Jul 29 22:05:15.568279 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.568239 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:05:15.570642 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.568474 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" podUID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" containerName="payload-pre-processing" containerID="cri-o://fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6" gracePeriod=30 Jul 29 22:05:15.829187 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.829129 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:05:15.907426 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.907384 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume\") pod \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " Jul 29 22:05:15.907593 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.907468 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qjdr\" (UniqueName: \"kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr\") pod \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\" (UID: \"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c\") " Jul 29 22:05:15.907849 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.907821 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" (UID: "a9e7c7c3-6f8e-4d8c-b07b-971153f6639c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:05:15.909856 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:15.909832 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr" (OuterVolumeSpecName: "kube-api-access-5qjdr") pod "a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" (UID: "a9e7c7c3-6f8e-4d8c-b07b-971153f6639c"). InnerVolumeSpecName "kube-api-access-5qjdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:05:16.008554 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.008506 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qjdr\" (UniqueName: \"kubernetes.io/projected/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-kube-api-access-5qjdr\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:05:16.008554 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.008543 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:05:16.356015 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.355976 2527 generic.go:356] "Generic (PLEG): container finished" podID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" containerID="fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6" exitCode=0 Jul 29 22:05:16.356015 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.356026 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" Jul 29 22:05:16.356231 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.356066 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" event={"ID":"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c","Type":"ContainerDied","Data":"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6"} Jul 29 22:05:16.356231 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.356122 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc" event={"ID":"a9e7c7c3-6f8e-4d8c-b07b-971153f6639c","Type":"ContainerDied","Data":"7fd8136efa2121aa907946a8282b274e05b43b050a76deb9a00a8a710758a9ad"} Jul 29 22:05:16.356231 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.356145 2527 scope.go:117] "RemoveContainer" containerID="fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6" Jul 29 22:05:16.364994 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.364972 2527 scope.go:117] "RemoveContainer" containerID="fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6" Jul 29 22:05:16.365261 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:05:16.365233 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6\": container with ID starting with fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6 not found: ID does not exist" containerID="fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6" Jul 29 22:05:16.365318 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.365270 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6"} err="failed to get container status \"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6\": rpc error: code = NotFound desc = could not find container \"fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6\": container with ID starting with fe96dd64baa42272e0aa4be92bf4d9b242c4365986eab3f43d3db809cc8ae8c6 not found: ID does not exist" Jul 29 22:05:16.382424 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.382390 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:05:16.386493 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.386468 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-6ab119-74745db744-nk8bc"] Jul 29 22:05:16.852764 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:16.852736 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" path="/var/lib/kubelet/pods/a9e7c7c3-6f8e-4d8c-b07b-971153f6639c/volumes" Jul 29 22:05:28.995526 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995491 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995760 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" containerName="payload-pre-processing" Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995771 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" containerName="payload-pre-processing" Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995777 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e20c02-8287-4fd3-b078-0bc214c598f5" containerName="revoke-keys" Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995784 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e20c02-8287-4fd3-b078-0bc214c598f5" containerName="revoke-keys" Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995836 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9e7c7c3-6f8e-4d8c-b07b-971153f6639c" containerName="payload-pre-processing" Jul 29 22:05:28.995921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:28.995842 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e20c02-8287-4fd3-b078-0bc214c598f5" containerName="revoke-keys" Jul 29 22:05:29.000051 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.000026 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.003502 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.003480 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-c9eccd\"" Jul 29 22:05:29.003625 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.003561 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-c9eccd-dockercfg-rps8d\"" Jul 29 22:05:29.020127 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.020101 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:05:29.115477 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.115443 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmx9v\" (UniqueName: \"kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.115477 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.115485 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.216236 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.216197 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmx9v\" (UniqueName: \"kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.216236 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.216241 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.216893 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.216875 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.225313 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.225286 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmx9v\" (UniqueName: \"kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v\") pod \"payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.310324 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.310229 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:29.444462 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:29.444354 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:05:29.447213 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:05:29.447184 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1474cf41_9f02_41ef_bd9a_e31bd366671a.slice/crio-95543ea201f085db23bfaf582a95bbea886d20234808f52b95a6f0883bdce612 WatchSource:0}: Error finding container 95543ea201f085db23bfaf582a95bbea886d20234808f52b95a6f0883bdce612: Status 404 returned error can't find the container with id 95543ea201f085db23bfaf582a95bbea886d20234808f52b95a6f0883bdce612 Jul 29 22:05:30.419821 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:30.419783 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" event={"ID":"1474cf41-9f02-41ef-bd9a-e31bd366671a","Type":"ContainerStarted","Data":"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69"} Jul 29 22:05:30.420280 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:30.419847 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" event={"ID":"1474cf41-9f02-41ef-bd9a-e31bd366671a","Type":"ContainerStarted","Data":"95543ea201f085db23bfaf582a95bbea886d20234808f52b95a6f0883bdce612"} Jul 29 22:05:30.420280 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:30.419945 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:30.439865 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:30.439813 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" podStartSLOduration=2.439796526 podStartE2EDuration="2.439796526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:30.437713801 +0000 UTC m=+2946.128145620" watchObservedRunningTime="2026-07-29 22:05:30.439796526 +0000 UTC m=+2946.130228343" Jul 29 22:05:41.426480 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:41.426450 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:05:58.155475 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.155391 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:05:58.158574 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.158557 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.160875 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.160850 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-ca895e\"" Jul 29 22:05:58.161035 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.161018 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-ca895e-dockercfg-xsfn2\"" Jul 29 22:05:58.164424 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.164402 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.164552 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.164439 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86m2\" (UniqueName: \"kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.176563 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.176538 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:05:58.265255 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.265210 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86m2\" (UniqueName: \"kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.265428 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.265297 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.266044 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.266023 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.288028 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.287990 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86m2\" (UniqueName: \"kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2\") pod \"payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.468085 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.467993 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:58.605559 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:58.605487 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:05:58.607777 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:05:58.607750 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f8b8a04_1b7e_4b25_9fa0_2d9bc8a1c125.slice/crio-6b9011ad60442a033b813474134ee161de0cd1ca04743726bdd35dd45e88e490 WatchSource:0}: Error finding container 6b9011ad60442a033b813474134ee161de0cd1ca04743726bdd35dd45e88e490: Status 404 returned error can't find the container with id 6b9011ad60442a033b813474134ee161de0cd1ca04743726bdd35dd45e88e490 Jul 29 22:05:59.548719 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:59.548680 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" event={"ID":"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125","Type":"ContainerStarted","Data":"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5"} Jul 29 22:05:59.549183 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:59.548737 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" event={"ID":"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125","Type":"ContainerStarted","Data":"6b9011ad60442a033b813474134ee161de0cd1ca04743726bdd35dd45e88e490"} Jul 29 22:05:59.549183 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:59.548799 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:05:59.568645 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:05:59.568566 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" podStartSLOduration=1.568546859 podStartE2EDuration="1.568546859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:05:59.567552781 +0000 UTC m=+2975.257984599" watchObservedRunningTime="2026-07-29 22:05:59.568546859 +0000 UTC m=+2975.258978677" Jul 29 22:06:10.556085 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:06:10.556048 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:06:24.966722 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:06:24.966693 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:06:24.971229 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:06:24.971205 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:06:24.973123 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:06:24.973104 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:06:24.977351 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:06:24.977335 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:09:25.137753 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:09:25.137725 2527 scope.go:117] "RemoveContainer" containerID="27a66719ca79afbb29df114288c7df54de6f637cec54b41cd0b9e5cd678d874f" Jul 29 22:09:25.145763 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:09:25.145742 2527 scope.go:117] "RemoveContainer" containerID="ed6bfb4b3b16a44adcc34410d1d21ad999030035e580e5e877ea6c7ac36092f4" Jul 29 22:11:24.992727 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:11:24.992694 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:11:24.995238 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:11:24.995219 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:11:24.999251 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:11:24.999230 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:11:25.001553 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:11:25.001537 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:11:25.172046 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:11:25.172017 2527 scope.go:117] "RemoveContainer" containerID="aebf8f5d27ddee14988a3256326e030b9a89a614c11d9c97cd82b5a1b7cb9880" Jul 29 22:14:37.606542 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.606507 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:14:37.609555 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.609535 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.612214 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.612193 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c9eccd\"/\"kube-root-ca.crt\"" Jul 29 22:14:37.612865 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.612848 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-c9eccd\"/\"openshift-service-ca.crt\"" Jul 29 22:14:37.612986 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.612851 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c9eccd\"/\"default-dockercfg-gkfjv\"" Jul 29 22:14:37.612986 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.612882 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-c9eccd\"/\"rate-test-model-c9eccd-kserve-self-signed-certs\"" Jul 29 22:14:37.619672 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.619646 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:14:37.660090 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660049 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.660090 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660093 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.660322 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660151 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.660322 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660208 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv5kt\" (UniqueName: \"kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.660322 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660245 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.660322 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.660261 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761415 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761381 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761415 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761419 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761637 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761448 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761637 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761492 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv5kt\" (UniqueName: \"kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761637 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761527 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761637 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761549 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761859 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761835 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.761948 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761876 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.762004 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.761944 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.763950 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.763903 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.764097 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.764073 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.770214 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.770191 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv5kt\" (UniqueName: \"kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt\") pod \"rate-test-model-c9eccd-kserve-79df76655-5xsrq\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:37.920050 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:37.919968 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:38.053955 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:38.053914 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:14:38.055994 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:14:38.055962 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c4c9002_02c5_44d3_ab16_bfd754b50ed7.slice/crio-92a2267d53475fd4913d89c9266dcf0aa79c74647ee7ab84623b1d1aed9b25df WatchSource:0}: Error finding container 92a2267d53475fd4913d89c9266dcf0aa79c74647ee7ab84623b1d1aed9b25df: Status 404 returned error can't find the container with id 92a2267d53475fd4913d89c9266dcf0aa79c74647ee7ab84623b1d1aed9b25df Jul 29 22:14:38.796743 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:38.796713 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" event={"ID":"7c4c9002-02c5-44d3-ab16-bfd754b50ed7","Type":"ContainerStarted","Data":"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4"} Jul 29 22:14:38.797185 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:38.796753 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" event={"ID":"7c4c9002-02c5-44d3-ab16-bfd754b50ed7","Type":"ContainerStarted","Data":"92a2267d53475fd4913d89c9266dcf0aa79c74647ee7ab84623b1d1aed9b25df"} Jul 29 22:14:38.816511 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:38.816456 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" podStartSLOduration=1.8164383800000001 podStartE2EDuration="1.81643838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:14:38.81525225 +0000 UTC m=+3494.505684068" watchObservedRunningTime="2026-07-29 22:14:38.81643838 +0000 UTC m=+3494.506870240" Jul 29 22:14:47.921049 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:47.920956 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:47.921049 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:47.921000 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:47.933285 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:47.933258 2527 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:14:48.853151 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:14:48.853124 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:16:25.018210 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:16:25.018086 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:16:25.023245 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:16:25.020896 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:16:25.025003 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:16:25.024979 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:16:25.027285 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:16:25.027265 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:16:52.827293 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:16:52.827250 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:17:07.846424 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:07.846384 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" podUID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" containerName="main" containerID="cri-o://298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4" gracePeriod=45 Jul 29 22:17:08.091540 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.091515 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:17:08.189597 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189545 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.189781 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189635 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv5kt\" (UniqueName: \"kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.189781 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189670 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.189781 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189711 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.189781 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189768 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.190015 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189818 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home\") pod \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\" (UID: \"7c4c9002-02c5-44d3-ab16-bfd754b50ed7\") " Jul 29 22:17:08.190077 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.189997 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:17:08.190077 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.190008 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:17:08.190188 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.190120 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home" (OuterVolumeSpecName: "home") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:17:08.192057 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.192025 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm" (OuterVolumeSpecName: "dshm") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 22:17:08.192168 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.192065 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt" (OuterVolumeSpecName: "kube-api-access-qv5kt") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "kube-api-access-qv5kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:17:08.192168 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.192083 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c4c9002-02c5-44d3-ab16-bfd754b50ed7" (UID: "7c4c9002-02c5-44d3-ab16-bfd754b50ed7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 22:17:08.291152 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291103 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv5kt\" (UniqueName: \"kubernetes.io/projected/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-kube-api-access-qv5kt\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.291152 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291148 2527 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-model-cache\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.291152 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291158 2527 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tmp-dir\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.291152 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291165 2527 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-tls-certs\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.291424 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291175 2527 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-home\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.291424 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.291184 2527 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c4c9002-02c5-44d3-ab16-bfd754b50ed7-dshm\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:17:08.442182 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.442099 2527 generic.go:356] "Generic (PLEG): container finished" podID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" containerID="298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4" exitCode=0 Jul 29 22:17:08.442182 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.442147 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" Jul 29 22:17:08.442377 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.442191 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" event={"ID":"7c4c9002-02c5-44d3-ab16-bfd754b50ed7","Type":"ContainerDied","Data":"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4"} Jul 29 22:17:08.442377 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.442239 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq" event={"ID":"7c4c9002-02c5-44d3-ab16-bfd754b50ed7","Type":"ContainerDied","Data":"92a2267d53475fd4913d89c9266dcf0aa79c74647ee7ab84623b1d1aed9b25df"} Jul 29 22:17:08.442377 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.442260 2527 scope.go:117] "RemoveContainer" containerID="298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4" Jul 29 22:17:08.451063 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.451049 2527 scope.go:117] "RemoveContainer" containerID="298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4" Jul 29 22:17:08.451331 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:17:08.451313 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4\": container with ID starting with 298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4 not found: ID does not exist" containerID="298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4" Jul 29 22:17:08.451384 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.451352 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4"} err="failed to get container status \"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4\": rpc error: code = NotFound desc = could not find container \"298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4\": container with ID starting with 298daa4b1da11afe31da261bf4a5fbd65aa65c17b1727491add97cf6ed257ac4 not found: ID does not exist" Jul 29 22:17:08.463140 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.463115 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:17:08.468739 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.468716 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-c9eccd/rate-test-model-c9eccd-kserve-79df76655-5xsrq"] Jul 29 22:17:08.852190 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:08.852162 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" path="/var/lib/kubelet/pods/7c4c9002-02c5-44d3-ab16-bfd754b50ed7/volumes" Jul 29 22:17:21.446847 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.446810 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:17:21.447260 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.447125 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" containerName="main" Jul 29 22:17:21.447260 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.447137 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" containerName="main" Jul 29 22:17:21.447260 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.447188 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c4c9002-02c5-44d3-ab16-bfd754b50ed7" containerName="main" Jul 29 22:17:21.450047 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.450029 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.452250 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.452228 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ee9d15-dockercfg-fctsr\"" Jul 29 22:17:21.452365 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.452263 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ee9d15\"" Jul 29 22:17:21.458761 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.458736 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:17:21.472672 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.472641 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:17:21.475905 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.475887 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.487365 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.487338 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:17:21.605741 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.605687 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.605971 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.605775 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.605971 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.605802 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vddd\" (UniqueName: \"kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.605971 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.605828 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzc4l\" (UniqueName: \"kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.706294 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.706183 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzc4l\" (UniqueName: \"kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.706294 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.706245 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.706555 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.706314 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.706555 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.706345 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vddd\" (UniqueName: \"kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.707030 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.707010 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.707085 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.707012 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.714404 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.714376 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzc4l\" (UniqueName: \"kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l\") pod \"payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.714559 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.714416 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vddd\" (UniqueName: \"kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd\") pod \"payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.761551 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.761515 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:21.786799 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.786769 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:21.909966 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.909905 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:17:21.913469 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:17:21.913427 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5903d94f_95a3_49bb_ab6c_ab3cfab158c9.slice/crio-d385bc219869ff3965f694d2def4bbecd98fcba23f4d1018bd11ee94f5768bc1 WatchSource:0}: Error finding container d385bc219869ff3965f694d2def4bbecd98fcba23f4d1018bd11ee94f5768bc1: Status 404 returned error can't find the container with id d385bc219869ff3965f694d2def4bbecd98fcba23f4d1018bd11ee94f5768bc1 Jul 29 22:17:21.942701 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:21.942676 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:17:21.946020 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:17:21.945996 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4c7b108_aba7_424a_bd9f_ccc7107c0957.slice/crio-793eb0b5898b9c50830ba05455f0827a7d45ccf4dea2308c4bd80b1d7fe0e70b WatchSource:0}: Error finding container 793eb0b5898b9c50830ba05455f0827a7d45ccf4dea2308c4bd80b1d7fe0e70b: Status 404 returned error can't find the container with id 793eb0b5898b9c50830ba05455f0827a7d45ccf4dea2308c4bd80b1d7fe0e70b Jul 29 22:17:22.503649 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.503610 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" event={"ID":"5903d94f-95a3-49bb-ab6c-ab3cfab158c9","Type":"ContainerStarted","Data":"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205"} Jul 29 22:17:22.504072 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.503671 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" event={"ID":"5903d94f-95a3-49bb-ab6c-ab3cfab158c9","Type":"ContainerStarted","Data":"d385bc219869ff3965f694d2def4bbecd98fcba23f4d1018bd11ee94f5768bc1"} Jul 29 22:17:22.504072 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.503702 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:22.505164 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.505137 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" event={"ID":"a4c7b108-aba7-424a-bd9f-ccc7107c0957","Type":"ContainerStarted","Data":"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c"} Jul 29 22:17:22.505291 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.505192 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" event={"ID":"a4c7b108-aba7-424a-bd9f-ccc7107c0957","Type":"ContainerStarted","Data":"793eb0b5898b9c50830ba05455f0827a7d45ccf4dea2308c4bd80b1d7fe0e70b"} Jul 29 22:17:22.505291 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.505225 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:22.521311 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.521238 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" podStartSLOduration=1.5212210229999998 podStartE2EDuration="1.521221023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:22.520383484 +0000 UTC m=+3658.210815304" watchObservedRunningTime="2026-07-29 22:17:22.521221023 +0000 UTC m=+3658.211652819" Jul 29 22:17:22.537329 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:22.537278 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" podStartSLOduration=1.537259557 podStartE2EDuration="1.537259557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:22.535378915 +0000 UTC m=+3658.225810734" watchObservedRunningTime="2026-07-29 22:17:22.537259557 +0000 UTC m=+3658.227691399" Jul 29 22:17:33.511469 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:33.511437 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:17:33.511889 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:33.511512 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:17:50.660538 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.660454 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:17:50.663768 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.663747 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.666062 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.666040 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b26bbb-dockercfg-2dfb5\"" Jul 29 22:17:50.666242 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.666223 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b26bbb\"" Jul 29 22:17:50.675608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.675585 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:17:50.687858 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.687828 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:17:50.695896 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.695866 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.702177 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.702146 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:17:50.749394 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.749355 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.749570 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.749413 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtchl\" (UniqueName: \"kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.749570 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.749483 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.749570 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.749533 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s5df\" (UniqueName: \"kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.850034 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.849996 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.850206 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.850045 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtchl\" (UniqueName: \"kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.850206 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.850075 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.850206 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.850100 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s5df\" (UniqueName: \"kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.850822 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.850801 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.850959 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.850917 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.859257 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.859232 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtchl\" (UniqueName: \"kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl\") pod \"payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:50.859409 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.859269 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s5df\" (UniqueName: \"kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df\") pod \"payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:50.974768 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:50.974679 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:51.008060 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.008017 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:51.131414 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.131373 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:17:51.132834 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:17:51.132804 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccd3945_9c2e_4913_ab44_2e91d43c0c29.slice/crio-bef2ec8be7cdc35adb75be143c8fd4c8ecf2a6c0006c7d489cabca1972e176de WatchSource:0}: Error finding container bef2ec8be7cdc35adb75be143c8fd4c8ecf2a6c0006c7d489cabca1972e176de: Status 404 returned error can't find the container with id bef2ec8be7cdc35adb75be143c8fd4c8ecf2a6c0006c7d489cabca1972e176de Jul 29 22:17:51.161059 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.161037 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:17:51.169046 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:17:51.169016 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd500726f_2fad_4dee_96c4_6b14d5645f5d.slice/crio-2dd55f141097949a1a0b65bb0225333f7b5bfa803f98633894b265cf31374386 WatchSource:0}: Error finding container 2dd55f141097949a1a0b65bb0225333f7b5bfa803f98633894b265cf31374386: Status 404 returned error can't find the container with id 2dd55f141097949a1a0b65bb0225333f7b5bfa803f98633894b265cf31374386 Jul 29 22:17:51.634339 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.634304 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" event={"ID":"d500726f-2fad-4dee-96c4-6b14d5645f5d","Type":"ContainerStarted","Data":"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6"} Jul 29 22:17:51.634526 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.634366 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" event={"ID":"d500726f-2fad-4dee-96c4-6b14d5645f5d","Type":"ContainerStarted","Data":"2dd55f141097949a1a0b65bb0225333f7b5bfa803f98633894b265cf31374386"} Jul 29 22:17:51.634526 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.634397 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:17:51.635993 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.635967 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" event={"ID":"0ccd3945-9c2e-4913-ab44-2e91d43c0c29","Type":"ContainerStarted","Data":"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3"} Jul 29 22:17:51.636110 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.636014 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" event={"ID":"0ccd3945-9c2e-4913-ab44-2e91d43c0c29","Type":"ContainerStarted","Data":"bef2ec8be7cdc35adb75be143c8fd4c8ecf2a6c0006c7d489cabca1972e176de"} Jul 29 22:17:51.636110 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.636042 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:17:51.651460 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.651412 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" podStartSLOduration=1.651395776 podStartE2EDuration="1.651395776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:51.650186055 +0000 UTC m=+3687.340617875" watchObservedRunningTime="2026-07-29 22:17:51.651395776 +0000 UTC m=+3687.341827594" Jul 29 22:17:51.666492 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:17:51.666444 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" podStartSLOduration=1.666428451 podStartE2EDuration="1.666428451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:17:51.66531677 +0000 UTC m=+3687.355748605" watchObservedRunningTime="2026-07-29 22:17:51.666428451 +0000 UTC m=+3687.356860329" Jul 29 22:18:02.643696 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:18:02.643662 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:18:02.644141 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:18:02.643755 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:18:21.059852 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:18:21.059814 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25_d500726f-2fad-4dee-96c4-6b14d5645f5d/payload-processing/0.log" Jul 29 22:19:16.216012 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:16.215890 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8_a4c7b108-aba7-424a-bd9f-ccc7107c0957/payload-processing/0.log" Jul 29 22:19:16.472512 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:16.472434 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25_d500726f-2fad-4dee-96c4-6b14d5645f5d/payload-processing/0.log" Jul 29 22:19:22.173279 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.173241 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:22.176597 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.176577 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.178870 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.178851 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-eafc3c\"" Jul 29 22:19:22.179508 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.179493 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-eafc3c-dockercfg-rjph5\"" Jul 29 22:19:22.189324 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.189302 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:22.314199 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.314155 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.314373 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.314215 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjqwt\" (UniqueName: \"kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.415620 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.415579 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.415620 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.415625 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjqwt\" (UniqueName: \"kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.416350 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.416328 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.424109 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.424052 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjqwt\" (UniqueName: \"kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt\") pod \"payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.487108 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.487069 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:22.626646 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:22.626618 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:22.629465 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:19:22.629430 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c91d9b_1a81_4774_9605_bd126d60af2c.slice/crio-57cfaa8571c3f89cd038ea1b35757a115d838eb11be4b18842011e7e26d49d55 WatchSource:0}: Error finding container 57cfaa8571c3f89cd038ea1b35757a115d838eb11be4b18842011e7e26d49d55: Status 404 returned error can't find the container with id 57cfaa8571c3f89cd038ea1b35757a115d838eb11be4b18842011e7e26d49d55 Jul 29 22:19:23.048647 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:23.048562 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" event={"ID":"e8c91d9b-1a81-4774-9605-bd126d60af2c","Type":"ContainerStarted","Data":"1a1f9cea91fc9a3f375ccea0c237294bcb6435e9050cd23c10e371fa0421162b"} Jul 29 22:19:23.048647 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:23.048614 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" event={"ID":"e8c91d9b-1a81-4774-9605-bd126d60af2c","Type":"ContainerStarted","Data":"57cfaa8571c3f89cd038ea1b35757a115d838eb11be4b18842011e7e26d49d55"} Jul 29 22:19:23.048824 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:23.048655 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:23.064203 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:23.064069 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" podStartSLOduration=1.0640495 podStartE2EDuration="1.0640495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:19:23.06352126 +0000 UTC m=+3778.753953078" watchObservedRunningTime="2026-07-29 22:19:23.0640495 +0000 UTC m=+3778.754481319" Jul 29 22:19:34.055952 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:34.055897 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:46.140191 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.140151 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm"] Jul 29 22:19:46.143337 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.143318 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:46.145498 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.145478 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 22:19:46.145595 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.145551 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxbps\"" Jul 29 22:19:46.150116 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.150088 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm"] Jul 29 22:19:46.211384 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.211346 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm\" (UID: \"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:46.311815 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.311778 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm\" (UID: \"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:46.312403 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.312385 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm\" (UID: \"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:46.454401 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.454305 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:46.591449 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:46.591415 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm"] Jul 29 22:19:46.594015 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:19:46.593987 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbdddc64_cd93_47d0_aed9_ee86e5bbe4ea.slice/crio-9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7 WatchSource:0}: Error finding container 9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7: Status 404 returned error can't find the container with id 9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7 Jul 29 22:19:47.169054 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:47.169018 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" event={"ID":"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea","Type":"ContainerStarted","Data":"fcd36c873c6824348500a86e85e6b52b8f2b17034b34570d44931c1043e00068"} Jul 29 22:19:47.169460 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:47.169071 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" event={"ID":"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea","Type":"ContainerStarted","Data":"9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7"} Jul 29 22:19:47.184854 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:47.184802 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" podStartSLOduration=1.184785909 podStartE2EDuration="1.184785909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 22:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:19:47.183787101 +0000 UTC m=+3802.874218920" watchObservedRunningTime="2026-07-29 22:19:47.184785909 +0000 UTC m=+3802.875217783" Jul 29 22:19:48.176042 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:48.176010 2527 generic.go:356] "Generic (PLEG): container finished" podID="fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" containerID="fcd36c873c6824348500a86e85e6b52b8f2b17034b34570d44931c1043e00068" exitCode=0 Jul 29 22:19:48.176436 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:48.176098 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" event={"ID":"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea","Type":"ContainerDied","Data":"fcd36c873c6824348500a86e85e6b52b8f2b17034b34570d44931c1043e00068"} Jul 29 22:19:49.313523 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:49.313498 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:49.339208 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:49.339178 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca\") pod \"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea\" (UID: \"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea\") " Jul 29 22:19:49.339558 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:49.339531 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" (UID: "fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:19:49.440301 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:49.440211 2527 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea-maas-api-service-ca\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:19:50.187797 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:50.187771 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" Jul 29 22:19:50.187985 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:50.187797 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm" event={"ID":"fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea","Type":"ContainerDied","Data":"9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7"} Jul 29 22:19:50.187985 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:50.187839 2527 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e85782633ae803ff9fb5a5e2ad6e3e14a4e64649b9c4f1e69fff239516fece7" Jul 29 22:19:51.970597 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:51.970568 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:51.971014 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:51.970769 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" podUID="e8c91d9b-1a81-4774-9605-bd126d60af2c" containerName="payload-pre-processing" containerID="cri-o://1a1f9cea91fc9a3f375ccea0c237294bcb6435e9050cd23c10e371fa0421162b" gracePeriod=30 Jul 29 22:19:52.108693 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.108656 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm"] Jul 29 22:19:52.114962 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.114252 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-eafc3c-fxnrm"] Jul 29 22:19:52.199277 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.199233 2527 generic.go:356] "Generic (PLEG): container finished" podID="e8c91d9b-1a81-4774-9605-bd126d60af2c" containerID="1a1f9cea91fc9a3f375ccea0c237294bcb6435e9050cd23c10e371fa0421162b" exitCode=0 Jul 29 22:19:52.199428 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.199302 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" event={"ID":"e8c91d9b-1a81-4774-9605-bd126d60af2c","Type":"ContainerDied","Data":"1a1f9cea91fc9a3f375ccea0c237294bcb6435e9050cd23c10e371fa0421162b"} Jul 29 22:19:52.239019 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.238989 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:52.365507 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.365479 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjqwt\" (UniqueName: \"kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt\") pod \"e8c91d9b-1a81-4774-9605-bd126d60af2c\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " Jul 29 22:19:52.365507 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.365515 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume\") pod \"e8c91d9b-1a81-4774-9605-bd126d60af2c\" (UID: \"e8c91d9b-1a81-4774-9605-bd126d60af2c\") " Jul 29 22:19:52.365988 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.365958 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e8c91d9b-1a81-4774-9605-bd126d60af2c" (UID: "e8c91d9b-1a81-4774-9605-bd126d60af2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:19:52.367769 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.367745 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt" (OuterVolumeSpecName: "kube-api-access-fjqwt") pod "e8c91d9b-1a81-4774-9605-bd126d60af2c" (UID: "e8c91d9b-1a81-4774-9605-bd126d60af2c"). InnerVolumeSpecName "kube-api-access-fjqwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:19:52.466540 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.466501 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjqwt\" (UniqueName: \"kubernetes.io/projected/e8c91d9b-1a81-4774-9605-bd126d60af2c-kube-api-access-fjqwt\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:19:52.466540 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.466532 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c91d9b-1a81-4774-9605-bd126d60af2c-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:19:52.852467 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:52.852426 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" path="/var/lib/kubelet/pods/fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea/volumes" Jul 29 22:19:53.205340 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:53.205264 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" Jul 29 22:19:53.205340 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:53.205288 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76" event={"ID":"e8c91d9b-1a81-4774-9605-bd126d60af2c","Type":"ContainerDied","Data":"57cfaa8571c3f89cd038ea1b35757a115d838eb11be4b18842011e7e26d49d55"} Jul 29 22:19:53.205911 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:53.205350 2527 scope.go:117] "RemoveContainer" containerID="1a1f9cea91fc9a3f375ccea0c237294bcb6435e9050cd23c10e371fa0421162b" Jul 29 22:19:53.220740 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:53.220709 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:53.224096 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:53.224063 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-eafc3c-7cfcdc5564-wpz76"] Jul 29 22:19:54.853326 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:19:54.853290 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c91d9b-1a81-4774-9605-bd126d60af2c" path="/var/lib/kubelet/pods/e8c91d9b-1a81-4774-9605-bd126d60af2c/volumes" Jul 29 22:20:17.947224 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:17.947187 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:20:17.947668 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:17.947428 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" podUID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" containerName="payload-processing" containerID="cri-o://8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c" gracePeriod=30 Jul 29 22:20:17.955796 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:17.955769 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:20:17.956097 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:17.956056 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" podUID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" containerName="payload-pre-processing" containerID="cri-o://6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205" gracePeriod=30 Jul 29 22:20:18.236005 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.235976 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:20:18.238766 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.238743 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:20:18.321661 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.321624 2527 generic.go:356] "Generic (PLEG): container finished" podID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" containerID="6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205" exitCode=0 Jul 29 22:20:18.321952 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.321692 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" Jul 29 22:20:18.321952 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.321704 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" event={"ID":"5903d94f-95a3-49bb-ab6c-ab3cfab158c9","Type":"ContainerDied","Data":"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205"} Jul 29 22:20:18.321952 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.321774 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8" event={"ID":"5903d94f-95a3-49bb-ab6c-ab3cfab158c9","Type":"ContainerDied","Data":"d385bc219869ff3965f694d2def4bbecd98fcba23f4d1018bd11ee94f5768bc1"} Jul 29 22:20:18.321952 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.321827 2527 scope.go:117] "RemoveContainer" containerID="6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205" Jul 29 22:20:18.323541 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.323507 2527 generic.go:356] "Generic (PLEG): container finished" podID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" containerID="8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c" exitCode=0 Jul 29 22:20:18.323660 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.323554 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" Jul 29 22:20:18.323660 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.323632 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" event={"ID":"a4c7b108-aba7-424a-bd9f-ccc7107c0957","Type":"ContainerDied","Data":"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c"} Jul 29 22:20:18.323789 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.323684 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8" event={"ID":"a4c7b108-aba7-424a-bd9f-ccc7107c0957","Type":"ContainerDied","Data":"793eb0b5898b9c50830ba05455f0827a7d45ccf4dea2308c4bd80b1d7fe0e70b"} Jul 29 22:20:18.331972 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.331949 2527 scope.go:117] "RemoveContainer" containerID="6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205" Jul 29 22:20:18.332270 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:20:18.332250 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205\": container with ID starting with 6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205 not found: ID does not exist" containerID="6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205" Jul 29 22:20:18.332562 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.332279 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205"} err="failed to get container status \"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205\": rpc error: code = NotFound desc = could not find container \"6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205\": container with ID starting with 6d0d93f9cacab2cd9a3df91e8a057f21673eedbac58e468c6879cda6b09f2205 not found: ID does not exist" Jul 29 22:20:18.332562 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.332297 2527 scope.go:117] "RemoveContainer" containerID="8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c" Jul 29 22:20:18.342159 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.342139 2527 scope.go:117] "RemoveContainer" containerID="8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c" Jul 29 22:20:18.342625 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:20:18.342599 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c\": container with ID starting with 8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c not found: ID does not exist" containerID="8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c" Jul 29 22:20:18.342686 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.342634 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c"} err="failed to get container status \"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c\": rpc error: code = NotFound desc = could not find container \"8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c\": container with ID starting with 8b026bad3c0443e332a9ee598fc107b2d31fb39d95040ede408dcbec1c61ca5c not found: ID does not exist" Jul 29 22:20:18.390978 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.390919 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzc4l\" (UniqueName: \"kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l\") pod \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " Jul 29 22:20:18.391139 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.391009 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume\") pod \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\" (UID: \"5903d94f-95a3-49bb-ab6c-ab3cfab158c9\") " Jul 29 22:20:18.391139 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.391040 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume\") pod \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " Jul 29 22:20:18.391139 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.391101 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vddd\" (UniqueName: \"kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd\") pod \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\" (UID: \"a4c7b108-aba7-424a-bd9f-ccc7107c0957\") " Jul 29 22:20:18.391413 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.391387 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5903d94f-95a3-49bb-ab6c-ab3cfab158c9" (UID: "5903d94f-95a3-49bb-ab6c-ab3cfab158c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:20:18.391483 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.391441 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a4c7b108-aba7-424a-bd9f-ccc7107c0957" (UID: "a4c7b108-aba7-424a-bd9f-ccc7107c0957"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:20:18.393332 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.393309 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l" (OuterVolumeSpecName: "kube-api-access-nzc4l") pod "5903d94f-95a3-49bb-ab6c-ab3cfab158c9" (UID: "5903d94f-95a3-49bb-ab6c-ab3cfab158c9"). InnerVolumeSpecName "kube-api-access-nzc4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:20:18.393332 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.393314 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd" (OuterVolumeSpecName: "kube-api-access-5vddd") pod "a4c7b108-aba7-424a-bd9f-ccc7107c0957" (UID: "a4c7b108-aba7-424a-bd9f-ccc7107c0957"). InnerVolumeSpecName "kube-api-access-5vddd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:20:18.491829 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.491727 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzc4l\" (UniqueName: \"kubernetes.io/projected/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-kube-api-access-nzc4l\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:20:18.491829 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.491767 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5903d94f-95a3-49bb-ab6c-ab3cfab158c9-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:20:18.491829 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.491776 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4c7b108-aba7-424a-bd9f-ccc7107c0957-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:20:18.491829 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.491785 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vddd\" (UniqueName: \"kubernetes.io/projected/a4c7b108-aba7-424a-bd9f-ccc7107c0957-kube-api-access-5vddd\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:20:18.650188 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.650157 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:20:18.653740 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.653715 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ee9d15-5c858fd47b-h8xr8"] Jul 29 22:20:18.663776 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.663754 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:20:18.667962 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.667921 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ee9d15-667fffd984-vg5t8"] Jul 29 22:20:18.853398 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.853368 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" path="/var/lib/kubelet/pods/5903d94f-95a3-49bb-ab6c-ab3cfab158c9/volumes" Jul 29 22:20:18.853712 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:20:18.853699 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" path="/var/lib/kubelet/pods/a4c7b108-aba7-424a-bd9f-ccc7107c0957/volumes" Jul 29 22:21:23.098648 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.098602 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:21:23.099217 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.098868 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" podUID="d500726f-2fad-4dee-96c4-6b14d5645f5d" containerName="payload-processing" containerID="cri-o://3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6" gracePeriod=30 Jul 29 22:21:23.106589 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.106552 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:21:23.106915 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.106798 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" podUID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" containerName="payload-pre-processing" containerID="cri-o://785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3" gracePeriod=30 Jul 29 22:21:23.378965 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.378915 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:21:23.390820 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.390794 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:21:23.518441 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518402 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume\") pod \"d500726f-2fad-4dee-96c4-6b14d5645f5d\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " Jul 29 22:21:23.518647 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518479 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume\") pod \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " Jul 29 22:21:23.518647 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518526 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtchl\" (UniqueName: \"kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl\") pod \"d500726f-2fad-4dee-96c4-6b14d5645f5d\" (UID: \"d500726f-2fad-4dee-96c4-6b14d5645f5d\") " Jul 29 22:21:23.518647 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518549 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s5df\" (UniqueName: \"kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df\") pod \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\" (UID: \"0ccd3945-9c2e-4913-ab44-2e91d43c0c29\") " Jul 29 22:21:23.518890 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518863 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d500726f-2fad-4dee-96c4-6b14d5645f5d" (UID: "d500726f-2fad-4dee-96c4-6b14d5645f5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:21:23.519019 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.518910 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ccd3945-9c2e-4913-ab44-2e91d43c0c29" (UID: "0ccd3945-9c2e-4913-ab44-2e91d43c0c29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:21:23.520752 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.520731 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df" (OuterVolumeSpecName: "kube-api-access-4s5df") pod "0ccd3945-9c2e-4913-ab44-2e91d43c0c29" (UID: "0ccd3945-9c2e-4913-ab44-2e91d43c0c29"). InnerVolumeSpecName "kube-api-access-4s5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:21:23.520825 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.520806 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl" (OuterVolumeSpecName: "kube-api-access-qtchl") pod "d500726f-2fad-4dee-96c4-6b14d5645f5d" (UID: "d500726f-2fad-4dee-96c4-6b14d5645f5d"). InnerVolumeSpecName "kube-api-access-qtchl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:21:23.614790 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.614694 2527 generic.go:356] "Generic (PLEG): container finished" podID="d500726f-2fad-4dee-96c4-6b14d5645f5d" containerID="3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6" exitCode=0 Jul 29 22:21:23.614790 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.614744 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" Jul 29 22:21:23.615042 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.614782 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" event={"ID":"d500726f-2fad-4dee-96c4-6b14d5645f5d","Type":"ContainerDied","Data":"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6"} Jul 29 22:21:23.615042 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.614839 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25" event={"ID":"d500726f-2fad-4dee-96c4-6b14d5645f5d","Type":"ContainerDied","Data":"2dd55f141097949a1a0b65bb0225333f7b5bfa803f98633894b265cf31374386"} Jul 29 22:21:23.615042 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.614870 2527 scope.go:117] "RemoveContainer" containerID="3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6" Jul 29 22:21:23.616281 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.616253 2527 generic.go:356] "Generic (PLEG): container finished" podID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" containerID="785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3" exitCode=0 Jul 29 22:21:23.616409 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.616307 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" Jul 29 22:21:23.616409 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.616306 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" event={"ID":"0ccd3945-9c2e-4913-ab44-2e91d43c0c29","Type":"ContainerDied","Data":"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3"} Jul 29 22:21:23.616517 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.616450 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s" event={"ID":"0ccd3945-9c2e-4913-ab44-2e91d43c0c29","Type":"ContainerDied","Data":"bef2ec8be7cdc35adb75be143c8fd4c8ecf2a6c0006c7d489cabca1972e176de"} Jul 29 22:21:23.619257 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.619234 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtchl\" (UniqueName: \"kubernetes.io/projected/d500726f-2fad-4dee-96c4-6b14d5645f5d-kube-api-access-qtchl\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:23.619257 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.619258 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s5df\" (UniqueName: \"kubernetes.io/projected/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-kube-api-access-4s5df\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:23.619413 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.619271 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d500726f-2fad-4dee-96c4-6b14d5645f5d-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:23.619413 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.619284 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ccd3945-9c2e-4913-ab44-2e91d43c0c29-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:23.625351 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.625328 2527 scope.go:117] "RemoveContainer" containerID="3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6" Jul 29 22:21:23.625714 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:21:23.625686 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6\": container with ID starting with 3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6 not found: ID does not exist" containerID="3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6" Jul 29 22:21:23.625821 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.625730 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6"} err="failed to get container status \"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6\": rpc error: code = NotFound desc = could not find container \"3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6\": container with ID starting with 3b919065e35c9db507b536978f837715d7fe5a7103a3df00970d2661164335b6 not found: ID does not exist" Jul 29 22:21:23.625821 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.625756 2527 scope.go:117] "RemoveContainer" containerID="785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3" Jul 29 22:21:23.634233 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.634213 2527 scope.go:117] "RemoveContainer" containerID="785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3" Jul 29 22:21:23.634560 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:21:23.634540 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3\": container with ID starting with 785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3 not found: ID does not exist" containerID="785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3" Jul 29 22:21:23.634625 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.634573 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3"} err="failed to get container status \"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3\": rpc error: code = NotFound desc = could not find container \"785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3\": container with ID starting with 785a0c6c4712d0d5b6cb9a1c5198b76e9199c6b4e56e9e79750d840e63bcdab3 not found: ID does not exist" Jul 29 22:21:23.639882 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.639854 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:21:23.643685 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.643655 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b26bbb-6d66dc54cb-vqb25"] Jul 29 22:21:23.655276 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.655243 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:21:23.659103 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:23.659071 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b26bbb-967486bdd-kh84s"] Jul 29 22:21:24.852342 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:24.852304 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" path="/var/lib/kubelet/pods/0ccd3945-9c2e-4913-ab44-2e91d43c0c29/volumes" Jul 29 22:21:24.852721 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:24.852651 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d500726f-2fad-4dee-96c4-6b14d5645f5d" path="/var/lib/kubelet/pods/d500726f-2fad-4dee-96c4-6b14d5645f5d/volumes" Jul 29 22:21:25.045242 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:25.045101 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:21:25.065919 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:25.048512 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:21:25.065919 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:25.057630 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:21:25.065919 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:25.061733 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-djvst_7c58c126-7301-46d4-963d-f5c15a7f8712/ovn-acl-logging/0.log" Jul 29 22:21:40.746527 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:40.746491 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:21:40.747170 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:40.746798 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" podUID="1474cf41-9f02-41ef-bd9a-e31bd366671a" containerName="payload-pre-processing" containerID="cri-o://50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69" gracePeriod=30 Jul 29 22:21:41.009497 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.009432 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:21:41.164681 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.164646 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mmx9v\" (UniqueName: \"kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v\") pod \"1474cf41-9f02-41ef-bd9a-e31bd366671a\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " Jul 29 22:21:41.164888 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.164720 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume\") pod \"1474cf41-9f02-41ef-bd9a-e31bd366671a\" (UID: \"1474cf41-9f02-41ef-bd9a-e31bd366671a\") " Jul 29 22:21:41.165187 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.165137 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1474cf41-9f02-41ef-bd9a-e31bd366671a" (UID: "1474cf41-9f02-41ef-bd9a-e31bd366671a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:21:41.167088 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.167070 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v" (OuterVolumeSpecName: "kube-api-access-mmx9v") pod "1474cf41-9f02-41ef-bd9a-e31bd366671a" (UID: "1474cf41-9f02-41ef-bd9a-e31bd366671a"). InnerVolumeSpecName "kube-api-access-mmx9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:21:41.265901 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.265805 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmx9v\" (UniqueName: \"kubernetes.io/projected/1474cf41-9f02-41ef-bd9a-e31bd366671a-kube-api-access-mmx9v\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:41.265901 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.265842 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1474cf41-9f02-41ef-bd9a-e31bd366671a-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:41.699330 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.699293 2527 generic.go:356] "Generic (PLEG): container finished" podID="1474cf41-9f02-41ef-bd9a-e31bd366671a" containerID="50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69" exitCode=0 Jul 29 22:21:41.699519 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.699342 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" Jul 29 22:21:41.699519 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.699384 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" event={"ID":"1474cf41-9f02-41ef-bd9a-e31bd366671a","Type":"ContainerDied","Data":"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69"} Jul 29 22:21:41.699519 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.699444 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n" event={"ID":"1474cf41-9f02-41ef-bd9a-e31bd366671a","Type":"ContainerDied","Data":"95543ea201f085db23bfaf582a95bbea886d20234808f52b95a6f0883bdce612"} Jul 29 22:21:41.699519 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.699466 2527 scope.go:117] "RemoveContainer" containerID="50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69" Jul 29 22:21:41.708281 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.708262 2527 scope.go:117] "RemoveContainer" containerID="50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69" Jul 29 22:21:41.708577 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:21:41.708558 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69\": container with ID starting with 50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69 not found: ID does not exist" containerID="50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69" Jul 29 22:21:41.708622 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.708587 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69"} err="failed to get container status \"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69\": rpc error: code = NotFound desc = could not find container \"50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69\": container with ID starting with 50b9f4a1fe4757cb3276b2c248f70deb65585de79dcc78c5327e5730d020de69 not found: ID does not exist" Jul 29 22:21:41.727719 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.727682 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:21:41.729167 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:41.729137 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-c9eccd-fc8b57d9b-lmr5n"] Jul 29 22:21:42.852116 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:42.852078 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1474cf41-9f02-41ef-bd9a-e31bd366671a" path="/var/lib/kubelet/pods/1474cf41-9f02-41ef-bd9a-e31bd366671a/volumes" Jul 29 22:21:58.671653 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:58.671613 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:21:58.672196 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:58.671856 2527 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" podUID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" containerName="payload-processing" containerID="cri-o://773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5" gracePeriod=30 Jul 29 22:21:58.930597 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:58.930521 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:21:59.006169 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.006129 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h86m2\" (UniqueName: \"kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2\") pod \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " Jul 29 22:21:59.006356 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.006217 2527 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume\") pod \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\" (UID: \"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125\") " Jul 29 22:21:59.006607 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.006568 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" (UID: "5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 22:21:59.008455 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.008427 2527 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2" (OuterVolumeSpecName: "kube-api-access-h86m2") pod "5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" (UID: "5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125"). InnerVolumeSpecName "kube-api-access-h86m2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 22:21:59.107400 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.107356 2527 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86m2\" (UniqueName: \"kubernetes.io/projected/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-kube-api-access-h86m2\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:59.107400 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.107393 2527 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125-plugins-config-volume\") on node \"ip-10-0-141-233.ec2.internal\" DevicePath \"\"" Jul 29 22:21:59.780391 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.780357 2527 generic.go:356] "Generic (PLEG): container finished" podID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" containerID="773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5" exitCode=0 Jul 29 22:21:59.780825 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.780410 2527 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" Jul 29 22:21:59.780825 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.780449 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" event={"ID":"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125","Type":"ContainerDied","Data":"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5"} Jul 29 22:21:59.780825 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.780512 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j" event={"ID":"5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125","Type":"ContainerDied","Data":"6b9011ad60442a033b813474134ee161de0cd1ca04743726bdd35dd45e88e490"} Jul 29 22:21:59.780825 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.780539 2527 scope.go:117] "RemoveContainer" containerID="773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5" Jul 29 22:21:59.791096 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.791064 2527 scope.go:117] "RemoveContainer" containerID="773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5" Jul 29 22:21:59.791705 ip-10-0-141-233 kubenswrapper[2527]: E0729 22:21:59.791682 2527 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5\": container with ID starting with 773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5 not found: ID does not exist" containerID="773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5" Jul 29 22:21:59.791794 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.791713 2527 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5"} err="failed to get container status \"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5\": rpc error: code = NotFound desc = could not find container \"773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5\": container with ID starting with 773050c195b291d6e44afa75c1bfa962f535ec3b36f0be1b9556f9e4ec02baa5 not found: ID does not exist" Jul 29 22:21:59.804855 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.804817 2527 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:21:59.809671 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:21:59.809638 2527 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-ca895e-54f48cfdf8-59t7j"] Jul 29 22:22:00.853422 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:00.853389 2527 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" path="/var/lib/kubelet/pods/5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125/volumes" Jul 29 22:22:13.637966 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:13.637914 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-mnxd4_e05fe207-19c8-471e-8c6e-14074c22b864/copy-manifests/0.log" Jul 29 22:22:13.645232 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:13.645203 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-mnxd4_e05fe207-19c8-471e-8c6e-14074c22b864/manager/0.log" Jul 29 22:22:17.272073 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:17.271982 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md5bk_36096a1e-645c-426a-a38a-a287073b1b50/discovery/0.log" Jul 29 22:22:17.385079 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:17.385040 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-787cd99557-2s4t2_8d358801-09e8-4796-bd37-812f2ad9ae85/kube-auth-proxy/0.log" Jul 29 22:22:17.978459 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:17.978425 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d985499-qqtv9_d5532478-f5c9-4103-82bf-32217f059c20/router/0.log" Jul 29 22:22:27.929155 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:27.929119 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kv7sc_b3743918-d652-44d4-a116-68a05a655027/global-pull-secret-syncer/0.log" Jul 29 22:22:28.034284 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:28.034245 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wq45j_f31b11f5-cefa-4201-ae2b-b088d025948a/konnectivity-agent/0.log" Jul 29 22:22:28.155567 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:28.155520 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-233.ec2.internal_9b454936a2c3aaec87ddb9b7f44eca43/haproxy/0.log" Jul 29 22:22:35.636704 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:35.636663 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-cc865447c-vcrtl_c65fdaa6-0445-425f-84f9-c7dde76130af/metrics-server/0.log" Jul 29 22:22:35.886900 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:35.886811 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vn7r_47e0e612-daa8-49e2-a556-b7418658969f/node-exporter/0.log" Jul 29 22:22:35.913593 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:35.913551 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vn7r_47e0e612-daa8-49e2-a556-b7418658969f/kube-rbac-proxy/0.log" Jul 29 22:22:35.937124 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:35.937096 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9vn7r_47e0e612-daa8-49e2-a556-b7418658969f/init-textfile/0.log" Jul 29 22:22:38.340643 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340607 2527 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb"] Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340871 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340882 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340890 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1474cf41-9f02-41ef-bd9a-e31bd366671a" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340895 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="1474cf41-9f02-41ef-bd9a-e31bd366671a" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340902 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d500726f-2fad-4dee-96c4-6b14d5645f5d" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340907 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="d500726f-2fad-4dee-96c4-6b14d5645f5d" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340917 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340922 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340958 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340966 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340977 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c91d9b-1a81-4774-9605-bd126d60af2c" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340984 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c91d9b-1a81-4774-9605-bd126d60af2c" containerName="payload-pre-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340991 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" containerName="revoke-keys" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.340996 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" containerName="revoke-keys" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341009 2527 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341014 2527 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341058 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f8b8a04-1b7e-4b25-9fa0-2d9bc8a1c125" containerName="payload-processing" Jul 29 22:22:38.341061 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341068 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="5903d94f-95a3-49bb-ab6c-ab3cfab158c9" containerName="payload-pre-processing" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341075 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ccd3945-9c2e-4913-ab44-2e91d43c0c29" containerName="payload-pre-processing" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341080 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c91d9b-1a81-4774-9605-bd126d60af2c" containerName="payload-pre-processing" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341086 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="1474cf41-9f02-41ef-bd9a-e31bd366671a" containerName="payload-pre-processing" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341092 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbdddc64-cd93-47d0-aed9-ee86e5bbe4ea" containerName="revoke-keys" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341097 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4c7b108-aba7-424a-bd9f-ccc7107c0957" containerName="payload-processing" Jul 29 22:22:38.341608 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.341101 2527 memory_manager.go:368] "RemoveStaleState removing state" podUID="d500726f-2fad-4dee-96c4-6b14d5645f5d" containerName="payload-processing" Jul 29 22:22:38.345453 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.345427 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.349139 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.349115 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqkkn\"/\"openshift-service-ca.crt\"" Jul 29 22:22:38.349139 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.349134 2527 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqkkn\"/\"kube-root-ca.crt\"" Jul 29 22:22:38.349348 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.349142 2527 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jqkkn\"/\"default-dockercfg-vhgbw\"" Jul 29 22:22:38.357908 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.357881 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb"] Jul 29 22:22:38.439058 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.439028 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/1.log" Jul 29 22:22:38.442430 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.442395 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7phzp\" (UniqueName: \"kubernetes.io/projected/624d8465-97cd-4801-a623-819768f19bb1-kube-api-access-7phzp\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.442636 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.442437 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-proc\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.442636 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.442501 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-podres\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.442636 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.442535 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-sys\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.442636 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.442563 2527 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-lib-modules\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.443793 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.443773 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-qgtcj_3d0ac419-cc5c-43e3-a3ea-c690b3743268/console-operator/2.log" Jul 29 22:22:38.543125 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543085 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7phzp\" (UniqueName: \"kubernetes.io/projected/624d8465-97cd-4801-a623-819768f19bb1-kube-api-access-7phzp\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543125 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543127 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-proc\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543164 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-podres\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543183 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-sys\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543207 2527 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-lib-modules\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543297 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-sys\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543307 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-proc\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543359 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543338 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-podres\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.543556 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.543419 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/624d8465-97cd-4801-a623-819768f19bb1-lib-modules\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.552254 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.552222 2527 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7phzp\" (UniqueName: \"kubernetes.io/projected/624d8465-97cd-4801-a623-819768f19bb1-kube-api-access-7phzp\") pod \"perf-node-gather-daemonset-wpgrb\" (UID: \"624d8465-97cd-4801-a623-819768f19bb1\") " pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.655041 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.655000 2527 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.789806 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.789781 2527 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb"] Jul 29 22:22:38.792522 ip-10-0-141-233 kubenswrapper[2527]: W0729 22:22:38.792494 2527 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod624d8465_97cd_4801_a623_819768f19bb1.slice/crio-00ae4aa022a0e26675a2aa4170d302a5a372bb80a1f97fc9ed4bb53cbc617734 WatchSource:0}: Error finding container 00ae4aa022a0e26675a2aa4170d302a5a372bb80a1f97fc9ed4bb53cbc617734: Status 404 returned error can't find the container with id 00ae4aa022a0e26675a2aa4170d302a5a372bb80a1f97fc9ed4bb53cbc617734 Jul 29 22:22:38.953977 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.953853 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" event={"ID":"624d8465-97cd-4801-a623-819768f19bb1","Type":"ContainerStarted","Data":"fc716fff44d3c9c3ab29e4d02c0c970f73f616a039bc98be080b77d92f7ff22e"} Jul 29 22:22:38.953977 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.953919 2527 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" event={"ID":"624d8465-97cd-4801-a623-819768f19bb1","Type":"ContainerStarted","Data":"00ae4aa022a0e26675a2aa4170d302a5a372bb80a1f97fc9ed4bb53cbc617734"} Jul 29 22:22:38.954221 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.954093 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:38.968250 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:38.968185 2527 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" podStartSLOduration=0.968169454 podStartE2EDuration="968.169454ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 22:22:38.967320198 +0000 UTC m=+3974.657752017" watchObservedRunningTime="2026-07-29 22:22:38.968169454 +0000 UTC m=+3974.658601271" Jul 29 22:22:40.590023 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:40.589985 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5zszr_e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4/dns/0.log" Jul 29 22:22:40.611705 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:40.611656 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5zszr_e4165d5f-2b62-4d2b-9b17-9fbc73d0dad4/kube-rbac-proxy/0.log" Jul 29 22:22:40.752302 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:40.752268 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxx4w_514a6c7e-c678-4089-8371-2f50ec8bd987/dns-node-resolver/0.log" Jul 29 22:22:41.369521 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:41.369485 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ztx7p_58046276-d71d-4df4-afeb-a5f37f66c360/node-ca/0.log" Jul 29 22:22:42.488876 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:42.488841 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md5bk_36096a1e-645c-426a-a38a-a287073b1b50/discovery/0.log" Jul 29 22:22:42.511378 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:42.511341 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-787cd99557-2s4t2_8d358801-09e8-4796-bd37-812f2ad9ae85/kube-auth-proxy/0.log" Jul 29 22:22:42.683129 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:42.683096 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d985499-qqtv9_d5532478-f5c9-4103-82bf-32217f059c20/router/0.log" Jul 29 22:22:43.575721 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:43.575693 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pw6fw_a838926f-fe57-4fde-9529-fbf598e51fdc/serve-healthcheck-canary/0.log" Jul 29 22:22:44.182535 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:44.182502 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvbkc_edf33de6-8251-4745-92be-4d8e9bbdf24d/kube-rbac-proxy/0.log" Jul 29 22:22:44.205170 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:44.205145 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvbkc_edf33de6-8251-4745-92be-4d8e9bbdf24d/exporter/0.log" Jul 29 22:22:44.227322 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:44.227294 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dvbkc_edf33de6-8251-4745-92be-4d8e9bbdf24d/extractor/0.log" Jul 29 22:22:44.968383 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:44.968353 2527 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jqkkn/perf-node-gather-daemonset-wpgrb" Jul 29 22:22:47.020329 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:47.020293 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-mnxd4_e05fe207-19c8-471e-8c6e-14074c22b864/manager/0.log" Jul 29 22:22:47.047070 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:47.047044 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b55745745-mnxd4_e05fe207-19c8-471e-8c6e-14074c22b864/copy-manifests/0.log" Jul 29 22:22:48.548096 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:48.548023 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5999ff5f6f-qg5p6_e975faba-e82e-4892-bfe0-26bbd9fdc237/manager/0.log" Jul 29 22:22:48.574406 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:48.574372 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-dq2vr_1344f346-30eb-45e7-930b-7fd42e7e7a2c/openshift-lws-operator/0.log" Jul 29 22:22:48.841126 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:48.841020 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md5bk_36096a1e-645c-426a-a38a-a287073b1b50/discovery/0.log" Jul 29 22:22:48.851496 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:48.851469 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-787cd99557-2s4t2_8d358801-09e8-4796-bd37-812f2ad9ae85/kube-auth-proxy/0.log" Jul 29 22:22:48.972086 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:48.972057 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d985499-qqtv9_d5532478-f5c9-4103-82bf-32217f059c20/router/0.log" Jul 29 22:22:50.480982 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:50.480952 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-md5bk_36096a1e-645c-426a-a38a-a287073b1b50/discovery/0.log" Jul 29 22:22:50.499967 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:50.499942 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-787cd99557-2s4t2_8d358801-09e8-4796-bd37-812f2ad9ae85/kube-auth-proxy/0.log" Jul 29 22:22:50.630568 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:50.630524 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66d985499-qqtv9_d5532478-f5c9-4103-82bf-32217f059c20/router/0.log" Jul 29 22:22:55.537361 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.537332 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/kube-multus-additional-cni-plugins/0.log" Jul 29 22:22:55.565677 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.565645 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/egress-router-binary-copy/0.log" Jul 29 22:22:55.594178 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.594151 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/cni-plugins/0.log" Jul 29 22:22:55.620187 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.620160 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/bond-cni-plugin/0.log" Jul 29 22:22:55.651249 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.651223 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/routeoverride-cni/0.log" Jul 29 22:22:55.683921 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.683889 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/whereabouts-cni-bincopy/0.log" Jul 29 22:22:55.719989 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.719961 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t5hk2_946d043a-d4b9-42bc-87e5-41db6c5d7fde/whereabouts-cni/0.log" Jul 29 22:22:55.767898 ip-10-0-141-233 kubenswrapper[2527]: I0729 22:22:55.767865 2527 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-fnxnn_b7e344e6-185e-4fb2-bbf9-c07db3b2e1b2/kube-multus/0.log"