Jul 10 16:34:32.260665 ip-10-0-131-107 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 10 16:34:32.260678 ip-10-0-131-107 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 10 16:34:32.260685 ip-10-0-131-107 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 10 16:34:32.260898 ip-10-0-131-107 systemd[1]: Failed to start Kubernetes Kubelet. Jul 10 16:34:42.340194 ip-10-0-131-107 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 10 16:34:42.340208 ip-10-0-131-107 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot b72bc13cb08a427882acfb9c50d479b2 -- Jul 10 16:36:50.174106 ip-10-0-131-107 systemd[1]: Starting Kubernetes Kubelet... Jul 10 16:36:50.690849 ip-10-0-131-107 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:50.690849 ip-10-0-131-107 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 10 16:36:50.690849 ip-10-0-131-107 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:50.690849 ip-10-0-131-107 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 10 16:36:50.690849 ip-10-0-131-107 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 10 16:36:50.692236 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.692140 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 10 16:36:50.697361 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697344 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.697361 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697362 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697366 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697372 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697377 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697380 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697383 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697387 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697390 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697393 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697396 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697399 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697402 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697405 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697408 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697411 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697413 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697416 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697419 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697422 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697425 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.697429 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697428 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697431 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697434 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697437 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697440 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697444 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697447 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697449 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697460 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697463 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697466 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697469 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697471 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697474 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697477 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697479 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697484 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697488 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697491 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.697947 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697495 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697497 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697501 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697503 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697506 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697509 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697511 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697514 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697517 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697519 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697522 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697525 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697528 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697530 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697534 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697537 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697540 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697543 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697545 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697548 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.698428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697551 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697554 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697556 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697559 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697561 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697564 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697567 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697570 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697572 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697575 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697579 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697582 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697585 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697587 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697591 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697595 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697599 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697601 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697604 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.698959 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697607 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697618 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697621 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697624 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697627 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697629 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697632 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697634 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697638 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697641 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697644 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697647 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697650 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697653 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697655 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697658 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697661 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697663 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697666 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.699466 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.697668 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698528 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698535 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698538 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698541 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698544 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698547 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698550 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698552 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698556 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698559 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698562 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698565 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698568 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698571 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698583 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698587 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698591 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698594 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698597 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.699988 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698600 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698603 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698606 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698609 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698612 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698615 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698617 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698620 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698623 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698626 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698628 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698631 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698634 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698637 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698639 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698642 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698645 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698649 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698652 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.700511 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698654 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698657 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698660 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698663 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698666 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698668 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698673 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698677 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698681 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698692 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698695 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698698 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698701 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698704 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698706 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698709 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698712 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698715 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698718 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.701024 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698720 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698723 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698725 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698728 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698731 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698735 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698738 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698741 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698743 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698747 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698749 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698752 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698755 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698758 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698761 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698764 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698767 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698770 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698773 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698776 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.701534 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698778 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698781 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698785 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698791 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698794 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698797 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698800 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698803 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698806 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698809 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698811 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698831 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698836 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698841 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698843 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698846 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698849 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698852 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698854 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698857 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.702065 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698862 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.698865 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699300 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699305 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699308 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699311 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699314 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699317 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699319 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699322 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699325 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699328 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699331 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699334 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699337 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699340 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699343 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699346 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699349 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699352 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.702588 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699354 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699357 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699360 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699363 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699365 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699368 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699371 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699374 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699377 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699382 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699385 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699388 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699391 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699394 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699397 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699400 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699403 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699406 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699408 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699411 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.703119 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699414 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699416 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699419 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699424 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699428 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699432 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699436 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699439 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699451 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699454 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699457 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699459 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699462 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699465 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699468 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699471 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699474 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699476 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699479 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.703634 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699483 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699486 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699489 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699492 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699495 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699498 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699501 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699504 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699507 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699510 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699513 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699516 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699519 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699521 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699524 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699527 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699529 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699532 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699534 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699537 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.704123 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699540 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699544 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699554 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699557 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699559 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699562 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699564 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699567 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699570 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699572 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699576 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699579 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699582 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699584 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699588 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699590 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699593 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699597 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699599 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699602 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.704674 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699605 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.699608 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699680 2568 flags.go:64] FLAG: --address="0.0.0.0" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699687 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699694 2568 flags.go:64] FLAG: --anonymous-auth="true" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699698 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699703 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699706 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699711 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699716 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699719 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699723 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699726 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699730 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699733 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699743 2568 flags.go:64] FLAG: --cgroup-root="" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699746 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699749 2568 flags.go:64] FLAG: --client-ca-file="" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699752 2568 flags.go:64] FLAG: --cloud-provider="external" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699755 2568 flags.go:64] FLAG: --cluster-dns="[]" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699762 2568 flags.go:64] FLAG: --cluster-domain="" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699766 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699769 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699772 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 10 16:36:50.705214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699775 2568 flags.go:64] FLAG: --container-log-max-files="5" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699779 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699783 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699786 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699790 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699793 2568 flags.go:64] FLAG: --contention-profiling="false" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699796 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699799 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699803 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699806 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699810 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699828 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699833 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699838 2568 flags.go:64] FLAG: --enable-load-reader="false" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699842 2568 flags.go:64] FLAG: --enable-server="true" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699845 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699850 2568 flags.go:64] FLAG: --event-burst="100" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699853 2568 flags.go:64] FLAG: --event-qps="50" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699856 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699860 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699863 2568 flags.go:64] FLAG: --eviction-hard="" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699867 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699870 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699873 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699878 2568 flags.go:64] FLAG: --eviction-soft="" Jul 10 16:36:50.705870 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699881 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699884 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699887 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699890 2568 flags.go:64] FLAG: --experimental-mounter-path="" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699893 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699896 2568 flags.go:64] FLAG: --fail-swap-on="true" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699899 2568 flags.go:64] FLAG: --feature-gates="" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699903 2568 flags.go:64] FLAG: --file-check-frequency="20s" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699906 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699910 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699914 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699917 2568 flags.go:64] FLAG: --healthz-port="10248" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699920 2568 flags.go:64] FLAG: --help="false" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699924 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-131-107.ec2.internal" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699927 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699930 2568 flags.go:64] FLAG: --http-check-frequency="20s" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699933 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699937 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699941 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699944 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699947 2568 flags.go:64] FLAG: --image-service-endpoint="" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699950 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699953 2568 flags.go:64] FLAG: --kube-api-burst="100" Jul 10 16:36:50.706523 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699956 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699959 2568 flags.go:64] FLAG: --kube-api-qps="50" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699962 2568 flags.go:64] FLAG: --kube-reserved="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699965 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699968 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699971 2568 flags.go:64] FLAG: --kubelet-cgroups="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699974 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699977 2568 flags.go:64] FLAG: --lock-file="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699980 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699984 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699987 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699993 2568 flags.go:64] FLAG: --log-json-split-stream="false" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699996 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.699999 2568 flags.go:64] FLAG: --log-text-split-stream="false" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700002 2568 flags.go:64] FLAG: --logging-format="text" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700005 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700009 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700012 2568 flags.go:64] FLAG: --manifest-url="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700018 2568 flags.go:64] FLAG: --manifest-url-header="" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700022 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700026 2568 flags.go:64] FLAG: --max-open-files="1000000" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700030 2568 flags.go:64] FLAG: --max-pods="110" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700033 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700037 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700040 2568 flags.go:64] FLAG: --memory-manager-policy="None" Jul 10 16:36:50.707166 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700043 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700046 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700049 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700052 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700060 2568 flags.go:64] FLAG: --node-status-max-images="50" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700063 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700066 2568 flags.go:64] FLAG: --oom-score-adj="-999" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700069 2568 flags.go:64] FLAG: --pod-cidr="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700072 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700078 2568 flags.go:64] FLAG: --pod-manifest-path="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700081 2568 flags.go:64] FLAG: --pod-max-pids="-1" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700084 2568 flags.go:64] FLAG: --pods-per-core="0" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700087 2568 flags.go:64] FLAG: --port="10250" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700091 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700094 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03ed9f45305ce5fe8" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700097 2568 flags.go:64] FLAG: --qos-reserved="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700101 2568 flags.go:64] FLAG: --read-only-port="10255" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700104 2568 flags.go:64] FLAG: --register-node="true" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700108 2568 flags.go:64] FLAG: --register-with-taints="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700111 2568 flags.go:64] FLAG: --registry-burst="10" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700114 2568 flags.go:64] FLAG: --registry-qps="5" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700117 2568 flags.go:64] FLAG: --reserved-cpus="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700120 2568 flags.go:64] FLAG: --reserved-memory="" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700129 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700132 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 10 16:36:50.707959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700137 2568 flags.go:64] FLAG: --rotate-certificates="false" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700140 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700143 2568 flags.go:64] FLAG: --runonce="false" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700146 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700151 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700154 2568 flags.go:64] FLAG: --seccomp-default="false" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700157 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700160 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700164 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700167 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700170 2568 flags.go:64] FLAG: --storage-driver-password="root" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700173 2568 flags.go:64] FLAG: --storage-driver-secure="false" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700177 2568 flags.go:64] FLAG: --storage-driver-table="stats" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700179 2568 flags.go:64] FLAG: --storage-driver-user="root" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700182 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700186 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700189 2568 flags.go:64] FLAG: --system-cgroups="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700191 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700197 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700200 2568 flags.go:64] FLAG: --tls-cert-file="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700203 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700207 2568 flags.go:64] FLAG: --tls-min-version="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700210 2568 flags.go:64] FLAG: --tls-private-key-file="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700213 2568 flags.go:64] FLAG: --topology-manager-policy="none" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700216 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 10 16:36:50.708630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700222 2568 flags.go:64] FLAG: --topology-manager-scope="container" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700225 2568 flags.go:64] FLAG: --v="2" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700229 2568 flags.go:64] FLAG: --version="false" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700234 2568 flags.go:64] FLAG: --vmodule="" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700237 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.700241 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700366 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700372 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700376 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700379 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700382 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700387 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700390 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700393 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700395 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700399 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700401 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700404 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700407 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700409 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700412 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.709305 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700415 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700418 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700421 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700423 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700426 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700429 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700431 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700434 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700438 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700443 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700446 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700449 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700453 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700456 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700459 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700462 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700465 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700468 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700472 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.709881 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700475 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700478 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700481 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700486 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700489 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700491 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700494 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700497 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700500 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700502 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700505 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700509 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700513 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700516 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700519 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700522 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700525 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700528 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700530 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700533 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.710374 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700536 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700538 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700541 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700544 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700546 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700549 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700556 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700559 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700561 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700564 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700567 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700572 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700575 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700578 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700580 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700584 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700587 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700589 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700593 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700595 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.710911 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700598 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700601 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700606 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700609 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700613 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700616 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700619 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700622 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700625 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700629 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700631 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700634 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700637 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700640 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700644 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700648 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700651 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700654 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700657 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.711428 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700660 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700663 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700666 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700668 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700672 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.700675 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.701273 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.708803 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.708846 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.708870 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.708877 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708944 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708950 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708954 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.711942 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708957 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708961 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708964 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708967 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708970 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708973 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708975 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708978 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708981 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708984 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708987 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708990 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708994 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708997 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.708999 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709002 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709005 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709008 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709017 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709020 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.712306 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709023 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709025 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709028 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709031 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709034 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709037 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709041 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709044 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709047 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709049 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709052 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709055 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709058 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709061 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709065 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709069 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709072 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709075 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709079 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.712839 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709083 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709086 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709089 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709092 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709095 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709098 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709101 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709104 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709107 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709110 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709113 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709115 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709124 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709129 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709132 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709135 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709138 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709142 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709146 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.713354 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709149 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709152 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709155 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709158 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709161 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709164 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709166 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709169 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709172 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709174 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709177 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709180 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709182 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709185 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709188 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709190 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709193 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709196 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709198 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.713935 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709201 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709203 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709206 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709210 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709213 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709216 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709219 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709222 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709225 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709228 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709231 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709235 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709238 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709241 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709244 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709246 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709249 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709251 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.714456 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709255 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.709260 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709382 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709387 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709390 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709395 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709399 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709402 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709406 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709409 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709411 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709414 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709417 2568 feature_gate.go:328] unrecognized feature gate: Example2 Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709420 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 10 16:36:50.714945 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709423 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709425 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709428 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709431 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709433 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709436 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709439 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709441 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709445 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709448 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709450 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709453 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709456 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709459 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709461 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709464 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709467 2568 feature_gate.go:328] unrecognized feature gate: Example Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709469 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709472 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709475 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 10 16:36:50.715328 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709477 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709480 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709483 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709485 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709488 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709490 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709493 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709496 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709498 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709501 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709504 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709507 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709509 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709512 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709515 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709518 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709520 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709523 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709526 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 10 16:36:50.715874 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709528 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709531 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709534 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709537 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709541 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709545 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709548 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709552 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709554 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709557 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709560 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709562 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709565 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709568 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709570 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709573 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709575 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709578 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709581 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 10 16:36:50.716362 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709584 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709587 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709589 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709592 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709595 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709598 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709601 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709603 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709606 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709609 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709611 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709614 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709617 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709620 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709622 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709625 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709628 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709631 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709634 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709636 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 10 16:36:50.716875 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709639 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709642 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709644 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709647 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709649 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709652 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709656 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709660 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:50.709663 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.709668 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.709787 2568 server.go:962] "Client rotation is on, will bootstrap in background" Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.714590 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.716105 2568 server.go:1019] "Starting client certificate rotation" Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.716216 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:36:50.717392 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.716258 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 10 16:36:50.719001 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.718986 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:36:50.724949 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.724926 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 10 16:36:50.742089 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.742063 2568 log.go:25] "Validated CRI v1 runtime API" Jul 10 16:36:50.744378 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.744364 2568 log.go:25] "Validated CRI v1 image API" Jul 10 16:36:50.745397 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.745378 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 10 16:36:50.746732 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.746714 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:36:50.748464 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.748433 2568 fs.go:135] Filesystem UUIDs: map[3cf23c21-0e94-41c9-a8f3-b134d4a618cf:/dev/nvme0n1p3 59e06015-f4a4-40e5-9c50-2d3dd4a02e2e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 10 16:36:50.748536 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.748466 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 10 16:36:50.754375 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.754261 2568 manager.go:216] Machine: {Timestamp:2026-07-10 16:36:50.752792305 +0000 UTC m=+0.428854082 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102548 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21094ee99b603336f6d7643ea6b855 SystemUUID:ec21094e-e99b-6033-36f6-d7643ea6b855 BootID:b72bc13c-b08a-4278-82ac-fb9c50d479b2 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ce:6c:40:18:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ce:6c:40:18:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:02:6c:f4:fc:df:b3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 10 16:36:50.754375 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.754366 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 10 16:36:50.754511 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.754454 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 10 16:36:50.757811 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.757777 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 10 16:36:50.757972 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.757825 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 10 16:36:50.758639 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.758629 2568 topology_manager.go:138] "Creating topology manager with none policy" Jul 10 16:36:50.758672 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.758643 2568 container_manager_linux.go:309] "Creating device plugin manager" Jul 10 16:36:50.758672 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.758650 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:36:50.758730 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.758675 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 10 16:36:50.759647 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.759636 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 10 16:36:50.762787 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.762774 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:36:50.762940 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.762929 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 10 16:36:50.762982 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.762964 2568 kubelet.go:486] "Attempting to sync node with API server" Jul 10 16:36:50.762982 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.762976 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 10 16:36:50.763041 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.762990 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 10 16:36:50.763041 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.763000 2568 kubelet.go:398] "Adding apiserver pod source" Jul 10 16:36:50.763041 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.763010 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 10 16:36:50.763426 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.763414 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:36:50.763476 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.763434 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 10 16:36:50.766073 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.766055 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 10 16:36:50.768214 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.768197 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 10 16:36:50.768297 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.768240 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 10 16:36:50.769088 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769073 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 10 16:36:50.769155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769094 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 10 16:36:50.769155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769103 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 10 16:36:50.769155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769112 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 10 16:36:50.769155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769124 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 10 16:36:50.769155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769132 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 10 16:36:50.769361 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769263 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 10 16:36:50.769361 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769314 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 10 16:36:50.769361 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769328 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 10 16:36:50.769361 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769339 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 10 16:36:50.769361 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769353 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 10 16:36:50.769527 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.769368 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 10 16:36:50.770466 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.770430 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 10 16:36:50.770466 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.770464 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 10 16:36:50.772800 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.772777 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qkrmk" Jul 10 16:36:50.773298 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.773277 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 10 16:36:50.773364 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.773303 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 10 16:36:50.773364 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.773300 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 10 16:36:50.775702 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.775682 2568 server.go:1268] "Started kubelet" Jul 10 16:36:50.775793 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.775762 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 10 16:36:50.776013 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.775956 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 10 16:36:50.776102 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.776054 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 10 16:36:50.776714 ip-10-0-131-107 systemd[1]: Started Kubernetes Kubelet. Jul 10 16:36:50.778149 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.778132 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 10 16:36:50.778373 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.778359 2568 server.go:310] "Adding debug handlers to kubelet server" Jul 10 16:36:50.778856 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.778837 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qkrmk" Jul 10 16:36:50.781370 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.781347 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 10 16:36:50.782187 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.780892 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-107.ec2.internal.18c0fafb8ea52c59 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-107.ec2.internal,UID:ip-10-0-131-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-107.ec2.internal,},FirstTimestamp:2026-07-10 16:36:50.775641177 +0000 UTC m=+0.451702955,LastTimestamp:2026-07-10 16:36:50.775641177 +0000 UTC m=+0.451702955,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-107.ec2.internal,}" Jul 10 16:36:50.785733 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.785718 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 10 16:36:50.785839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.785741 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 10 16:36:50.785991 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.785970 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:50.786589 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786484 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 10 16:36:50.786589 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786485 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 10 16:36:50.786589 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786514 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 10 16:36:50.786754 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786636 2568 reconstruct.go:97] "Volume reconstruction finished" Jul 10 16:36:50.786754 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786646 2568 reconciler.go:29] "Reconciler: start to sync state" Jul 10 16:36:50.786754 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786700 2568 factory.go:55] Registering systemd factory Jul 10 16:36:50.786754 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786715 2568 factory.go:223] Registration of the systemd container factory successfully Jul 10 16:36:50.786991 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786977 2568 factory.go:153] Registering CRI-O factory Jul 10 16:36:50.786991 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.786992 2568 factory.go:223] Registration of the crio container factory successfully Jul 10 16:36:50.787078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.787033 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 10 16:36:50.787078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.787058 2568 factory.go:103] Registering Raw factory Jul 10 16:36:50.787078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.787071 2568 manager.go:1196] Started watching for new ooms in manager Jul 10 16:36:50.787411 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.787393 2568 manager.go:349] Starting recovery of all containers Jul 10 16:36:50.796275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.796095 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:50.797030 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.796990 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 10 16:36:50.799188 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.799068 2568 manager.go:354] Recovery completed Jul 10 16:36:50.799278 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.799111 2568 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-131-107.ec2.internal\" not found" node="ip-10-0-131-107.ec2.internal" Jul 10 16:36:50.802341 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.802237 2568 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 10 16:36:50.806370 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.806355 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:50.808645 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.808628 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:50.808719 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.808661 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:50.808719 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.808675 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:50.809294 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.809279 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 10 16:36:50.809294 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.809293 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 10 16:36:50.809386 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.809311 2568 state_mem.go:36] "Initialized new in-memory state store" Jul 10 16:36:50.809386 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.809310 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:50.810306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810286 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:50.810409 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810326 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:50.810409 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810343 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:50.810745 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810733 2568 policy_none.go:49] "None policy: Start" Jul 10 16:36:50.810801 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810750 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 10 16:36:50.810801 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.810761 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 10 16:36:50.812132 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.812121 2568 policy_none.go:47] "Start" Jul 10 16:36:50.849432 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.849411 2568 manager.go:337] "Starting Device Plugin manager" Jul 10 16:36:50.849533 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.849447 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 10 16:36:50.849533 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.849458 2568 server.go:87] "Starting device plugin registration server" Jul 10 16:36:50.849774 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.849761 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 10 16:36:50.849839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.849776 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 10 16:36:50.850023 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.849922 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 10 16:36:50.850081 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.850027 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 10 16:36:50.850081 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.850036 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 10 16:36:50.850615 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.850597 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 10 16:36:50.850682 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.850654 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:50.950953 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.950861 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:50.952207 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.952146 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:50.952207 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.952180 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:50.952207 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.952193 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:50.952341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.952223 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-107.ec2.internal" Jul 10 16:36:50.961053 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:50.961034 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-107.ec2.internal" Jul 10 16:36:50.961155 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.961062 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-107.ec2.internal\": node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:50.978938 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:50.978909 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.037081 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.037056 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 10 16:36:51.037171 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.037098 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 10 16:36:51.037171 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.037122 2568 kubelet.go:2448] "Starting kubelet main sync loop" Jul 10 16:36:51.037171 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.037153 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 10 16:36:51.039625 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.039600 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:51.079774 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.079721 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.140605 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.140557 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal"] Jul 10 16:36:51.140687 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.140658 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:51.141810 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.141789 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:51.141938 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.141822 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:51.141938 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.141839 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:51.143326 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.143313 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:51.143466 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.143438 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.143513 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.143484 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:51.144110 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144088 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:51.144208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144091 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:51.144208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144148 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:51.144208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144160 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:51.144208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144123 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:51.144357 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.144210 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:51.145508 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.145491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.145570 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.145527 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 10 16:36:51.146246 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.146227 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientMemory" Jul 10 16:36:51.146331 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.146262 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasNoDiskPressure" Jul 10 16:36:51.146331 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.146278 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeHasSufficientPID" Jul 10 16:36:51.166505 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.166484 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-107.ec2.internal\" not found" node="ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.171033 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.171017 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-107.ec2.internal\" not found" node="ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.180470 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.180456 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.189070 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.189051 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44bd8c551779de4c442e1d9ae5fc2be1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-107.ec2.internal\" (UID: \"44bd8c551779de4c442e1d9ae5fc2be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.189143 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.189085 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.189143 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.189113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.281268 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.281194 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.290207 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44bd8c551779de4c442e1d9ae5fc2be1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-107.ec2.internal\" (UID: \"44bd8c551779de4c442e1d9ae5fc2be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.290272 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290213 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.290272 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290239 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.290350 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.290350 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44bd8c551779de4c442e1d9ae5fc2be1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-107.ec2.internal\" (UID: \"44bd8c551779de4c442e1d9ae5fc2be1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.290350 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.290310 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4b29e5d3cfbcc3884d91f6b4f3cf71c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal\" (UID: \"f4b29e5d3cfbcc3884d91f6b4f3cf71c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.388952 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.388902 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.475931 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.475895 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.476046 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.475895 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:51.489869 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.489848 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.590029 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.589934 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.687032 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.686996 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:51.690168 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.690147 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.715078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.715051 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 10 16:36:51.715581 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.715208 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:36:51.715581 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.715239 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:36:51.715581 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.715250 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 10 16:36:51.782609 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.782552 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-09 16:31:51 +0000 UTC" deadline="2028-01-08 07:53:38.989185105 +0000 UTC" Jul 10 16:36:51.782609 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:51.782598 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13119h16m47.206591375s" Jul 10 16:36:51.791318 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.791294 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.892234 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.892137 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:51.992904 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:51.992870 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-107.ec2.internal\" not found" Jul 10 16:36:52.057275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.057245 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:52.076186 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:52.076138 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b29e5d3cfbcc3884d91f6b4f3cf71c.slice/crio-bc0940807454b884c52444365723dc5867af7474dc5842bd2c8111e059379b7d WatchSource:0}: Error finding container bc0940807454b884c52444365723dc5867af7474dc5842bd2c8111e059379b7d: Status 404 returned error can't find the container with id bc0940807454b884c52444365723dc5867af7474dc5842bd2c8111e059379b7d Jul 10 16:36:52.079621 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.079602 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:36:52.086641 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.086622 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" Jul 10 16:36:52.099667 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.099647 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:52.100839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.100824 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" Jul 10 16:36:52.113817 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.113796 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 10 16:36:52.533691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.533657 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:52.539325 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:52.539296 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44bd8c551779de4c442e1d9ae5fc2be1.slice/crio-627b59866197b21798163f99029013c636ccbd0e5d8e125a3a2af556c6c44c85 WatchSource:0}: Error finding container 627b59866197b21798163f99029013c636ccbd0e5d8e125a3a2af556c6c44c85: Status 404 returned error can't find the container with id 627b59866197b21798163f99029013c636ccbd0e5d8e125a3a2af556c6c44c85 Jul 10 16:36:52.765223 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.765184 2568 apiserver.go:52] "Watching apiserver" Jul 10 16:36:52.776569 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.776536 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 10 16:36:52.776966 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.776937 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-nhcft","kube-system/konnectivity-agent-xmdzd","kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv","openshift-dns/node-resolver-2ndkl","openshift-image-registry/node-ca-692mv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal","openshift-multus/multus-zr4fb","openshift-cluster-node-tuning-operator/tuned-6nm5r","openshift-multus/multus-additional-cni-plugins-r276w","openshift-multus/network-metrics-daemon-8mssf","openshift-network-diagnostics/network-check-target-59mt6","openshift-network-operator/iptables-alerter-tlpg5","openshift-ovn-kubernetes/ovnkube-node-scsl6"] Jul 10 16:36:52.778561 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.778532 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.780243 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.780223 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.781586 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781542 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 10 16:36:52.781586 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781563 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 10 16:36:52.781586 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781578 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 10 16:36:52.781866 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781706 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.781939 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.782003 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.781982 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.782060 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.782011 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s7zq2\"" Jul 10 16:36:52.783363 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.782726 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.783363 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.782729 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-c45jp\"" Jul 10 16:36:52.783363 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.782888 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.783363 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.783002 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 10 16:36:52.783363 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.783335 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.784893 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.784706 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 10 16:36:52.784893 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.784751 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.785035 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 10 16:36:52.785092 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.785215 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785196 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 10 16:36:52.785387 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785279 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 10 16:36:52.785694 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785535 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.785694 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.785569 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-2xp4r\"" Jul 10 16:36:52.786259 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786131 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.786259 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786131 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 10 16:36:52.786427 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786346 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.786720 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9d224\"" Jul 10 16:36:52.786720 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786584 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 10 16:36:52.786888 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.786740 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.787389 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.787372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.787699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.787644 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.787971 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.787955 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-flhsz\"" Jul 10 16:36:52.788571 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.788519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.789411 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.789174 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 10 16:36:52.789522 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.789439 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bx8rx\"" Jul 10 16:36:52.789522 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.789494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.789522 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.789520 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.789972 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.789953 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.790949 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.790927 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:36:52.791047 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.790957 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sclwt\"" Jul 10 16:36:52.791047 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.790996 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 10 16:36:52.792092 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.792071 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:52.792574 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.792554 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6ggs4\"" Jul 10 16:36:52.792792 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.792769 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 10 16:36:52.793765 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.793728 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:52.793862 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:52.793837 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:36:52.794660 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.794624 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-g2jls\"" Jul 10 16:36:52.795084 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.794926 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 10 16:36:52.795084 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.795006 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 10 16:36:52.795295 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.795249 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:52.795359 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:52.795303 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:36:52.797398 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.797376 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:52.797482 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:52.797454 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.797873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-system-cni-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.797917 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-socket-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.797958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cnibin\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.797991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-os-release\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798023 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798056 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-slash\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.798087 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798089 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-sys-fs\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999c6\" (UniqueName: \"kubernetes.io/projected/ff236cfe-018a-42db-a189-3b1d84b276cf-kube-api-access-999c6\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798156 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3758299-2838-43fd-ad6a-bc8b885579cd-serviceca\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798202 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798272 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76d47f55-7101-4301-add0-569f73b6e8e7-host-slash\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-systemd-units\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-var-lib-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.798502 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-env-overrides\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798786 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjbs\" (UniqueName: \"kubernetes.io/projected/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-kube-api-access-9gjbs\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798836 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mkgh\" (UniqueName: \"kubernetes.io/projected/76d47f55-7101-4301-add0-569f73b6e8e7-kube-api-access-8mkgh\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798870 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-kubelet\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-netns\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-log-socket\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799007 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.798974 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-bin\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-config\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-registration-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq595\" (UniqueName: \"kubernetes.io/projected/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-kube-api-access-wq595\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799237 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799277 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-device-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr59t\" (UniqueName: \"kubernetes.io/projected/489aa976-73ae-42ea-9c9c-c2490cab6607-kube-api-access-vr59t\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-netd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799386 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76d47f55-7101-4301-add0-569f73b6e8e7-iptables-alerter-script\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799431 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-systemd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-node-log\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799493 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.799540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff236cfe-018a-42db-a189-3b1d84b276cf-hosts-file\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3758299-2838-43fd-ad6a-bc8b885579cd-host\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-ovn\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovn-node-metrics-cert\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-script-lib\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799861 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgxf\" (UniqueName: \"kubernetes.io/projected/a3758299-2838-43fd-ad6a-bc8b885579cd-kube-api-access-rlgxf\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799893 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-etc-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.799983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.799940 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff236cfe-018a-42db-a189-3b1d84b276cf-tmp-dir\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.821500 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.821473 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g88zv" Jul 10 16:36:52.829312 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.829291 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g88zv" Jul 10 16:36:52.888104 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.888073 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 10 16:36:52.901178 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:52.901341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901200 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-k8s-cni-cncf-io\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.901341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.901341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901311 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-systemd-units\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901429 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901431 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-systemd-units\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901485 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjbs\" (UniqueName: \"kubernetes.io/projected/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-kube-api-access-9gjbs\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-system-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901549 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-run\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901572 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-kubelet\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901589 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-log-socket\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-config\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901634 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-kubelet-config\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-kubernetes\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-sys\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901686 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-host\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.901717 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e0ea746-b225-44d5-9981-db1ecfc29ca5-konnectivity-ca\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901739 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-socket-dir-parent\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901784 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq595\" (UniqueName: \"kubernetes.io/projected/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-kube-api-access-wq595\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr59t\" (UniqueName: \"kubernetes.io/projected/489aa976-73ae-42ea-9c9c-c2490cab6607-kube-api-access-vr59t\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901867 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-cnibin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.901944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-os-release\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902009 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-log-socket\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-kubelet\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgxf\" (UniqueName: \"kubernetes.io/projected/a3758299-2838-43fd-ad6a-bc8b885579cd-kube-api-access-rlgxf\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902103 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-netd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-systemd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902178 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-systemd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902194 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-modprobe-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902233 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902295 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-netd\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902317 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-ovn\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovn-node-metrics-cert\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902358 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-ovn\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-script-lib\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-multus-daemon-config\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902454 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-config\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-conf\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902536 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff236cfe-018a-42db-a189-3b1d84b276cf-tmp-dir\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902574 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-dbus\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902607 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902635 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysconfig\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-tuned\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.902880 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-socket-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-bin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902752 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902778 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-multus\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902804 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-var-lib-kubelet\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cnibin\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff236cfe-018a-42db-a189-3b1d84b276cf-tmp-dir\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-os-release\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902853 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-socket-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999c6\" (UniqueName: \"kubernetes.io/projected/ff236cfe-018a-42db-a189-3b1d84b276cf-kube-api-access-999c6\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3758299-2838-43fd-ad6a-bc8b885579cd-serviceca\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902965 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-os-release\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.902985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-kubelet\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cnibin\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903010 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-lib-modules\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903010 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovnkube-script-lib\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.903691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903079 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76d47f55-7101-4301-add0-569f73b6e8e7-host-slash\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903124 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-var-lib-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903168 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-var-lib-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-env-overrides\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903193 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/76d47f55-7101-4301-add0-569f73b6e8e7-host-slash\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-hostroot\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e0ea746-b225-44d5-9981-db1ecfc29ca5-agent-certs\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903289 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mkgh\" (UniqueName: \"kubernetes.io/projected/76d47f55-7101-4301-add0-569f73b6e8e7-kube-api-access-8mkgh\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-netns\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-bin\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-registration-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-netns\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a3758299-2838-43fd-ad6a-bc8b885579cd-serviceca\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903419 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-etc-kubernetes\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903471 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7ss\" (UniqueName: \"kubernetes.io/projected/70523d1b-ee7e-495d-a301-ade04b748746-kube-api-access-fj7ss\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903456 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-cni-bin\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.904493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903497 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-registration-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-tmp\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903636 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-device-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903682 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903693 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-multus-certs\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903608 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-env-overrides\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903709 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-run-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903721 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-device-dir\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ckp\" (UniqueName: \"kubernetes.io/projected/cadf1967-b262-4bdd-a933-968447f88cf1-kube-api-access-s4ckp\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903754 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76d47f55-7101-4301-add0-569f73b6e8e7-iptables-alerter-script\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-node-log\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.905203 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.903959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff236cfe-018a-42db-a189-3b1d84b276cf-hosts-file\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-node-log\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904013 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff236cfe-018a-42db-a189-3b1d84b276cf-hosts-file\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3758299-2838-43fd-ad6a-bc8b885579cd-host\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-cni-binary-copy\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwj8\" (UniqueName: \"kubernetes.io/projected/f2f96e82-4aa4-48c0-8386-d629ce669284-kube-api-access-5kwj8\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-etc-selinux\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904125 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a3758299-2838-43fd-ad6a-bc8b885579cd-host\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904156 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904186 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-cni-binary-copy\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-netns\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904253 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-conf-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-etc-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904298 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-run-ovn-kubernetes\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904308 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-system-cni-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904326 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-etc-openvswitch\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904344 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-system-cni-dir\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.905916 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904343 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-systemd\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:52.906695 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-slash\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.906695 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/76d47f55-7101-4301-add0-569f73b6e8e7-iptables-alerter-script\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.906695 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-sys-fs\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.906695 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904461 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/489aa976-73ae-42ea-9c9c-c2490cab6607-sys-fs\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.906695 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.904462 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-host-slash\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.906918 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.906729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-ovn-node-metrics-cert\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.911576 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.911502 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjbs\" (UniqueName: \"kubernetes.io/projected/9bdf848b-cc24-4998-bb53-4a3ea919e2f6-kube-api-access-9gjbs\") pod \"ovnkube-node-scsl6\" (UID: \"9bdf848b-cc24-4998-bb53-4a3ea919e2f6\") " pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:52.912371 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.912348 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-999c6\" (UniqueName: \"kubernetes.io/projected/ff236cfe-018a-42db-a189-3b1d84b276cf-kube-api-access-999c6\") pod \"node-resolver-2ndkl\" (UID: \"ff236cfe-018a-42db-a189-3b1d84b276cf\") " pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:52.912483 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.912389 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mkgh\" (UniqueName: \"kubernetes.io/projected/76d47f55-7101-4301-add0-569f73b6e8e7-kube-api-access-8mkgh\") pod \"iptables-alerter-tlpg5\" (UID: \"76d47f55-7101-4301-add0-569f73b6e8e7\") " pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:52.912861 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.912817 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq595\" (UniqueName: \"kubernetes.io/projected/ec6e9897-b9bd-4f00-acbd-72aea3e101e7-kube-api-access-wq595\") pod \"multus-additional-cni-plugins-r276w\" (UID: \"ec6e9897-b9bd-4f00-acbd-72aea3e101e7\") " pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:52.913425 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.913405 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr59t\" (UniqueName: \"kubernetes.io/projected/489aa976-73ae-42ea-9c9c-c2490cab6607-kube-api-access-vr59t\") pod \"aws-ebs-csi-driver-node-tbszv\" (UID: \"489aa976-73ae-42ea-9c9c-c2490cab6607\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:52.914037 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:52.914021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgxf\" (UniqueName: \"kubernetes.io/projected/a3758299-2838-43fd-ad6a-bc8b885579cd-kube-api-access-rlgxf\") pod \"node-ca-692mv\" (UID: \"a3758299-2838-43fd-ad6a-bc8b885579cd\") " pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:53.005377 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005345 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-multus-certs\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005384 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ckp\" (UniqueName: \"kubernetes.io/projected/cadf1967-b262-4bdd-a933-968447f88cf1-kube-api-access-s4ckp\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005437 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwj8\" (UniqueName: \"kubernetes.io/projected/f2f96e82-4aa4-48c0-8386-d629ce669284-kube-api-access-5kwj8\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-cni-binary-copy\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005476 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-multus-certs\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-netns\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-conf-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.005555 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.005556 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-systemd\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.005628 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:53.505603472 +0000 UTC m=+3.181665253 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-k8s-cni-cncf-io\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.005684 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005686 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-system-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-run\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.005749 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:53.505731963 +0000 UTC m=+3.181793732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005783 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-run\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005801 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-kubelet-config\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005829 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-kubernetes\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-netns\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005848 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-systemd\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-sys\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-host\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005902 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-system-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005911 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e0ea746-b225-44d5-9981-db1ecfc29ca5-konnectivity-ca\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005939 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-socket-dir-parent\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005944 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-run-k8s-cni-cncf-io\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005793 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-conf-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005965 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-cnibin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.005992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-os-release\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-kubelet-config\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006020 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-modprobe-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-multus-daemon-config\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006114 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006119 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-cni-binary-copy\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-socket-dir-parent\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006141 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-conf\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006175 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-dbus\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-host\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-os-release\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006234 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-sys\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.006845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysconfig\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-kubernetes\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006269 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-tuned\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006140 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-modprobe-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-bin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-multus\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-var-lib-kubelet\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006363 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-conf\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-kubelet\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006406 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/29764102-4050-48c5-bc52-cc0e03dfc516-dbus\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-lib-modules\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006425 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysctl-d\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006453 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-multus\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-cni-bin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006292 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-cnibin\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-host-var-lib-kubelet\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006527 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-hostroot\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.007779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006454 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-sysconfig\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e0ea746-b225-44d5-9981-db1ecfc29ca5-konnectivity-ca\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006580 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-multus-cni-dir\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-hostroot\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006613 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e0ea746-b225-44d5-9981-db1ecfc29ca5-agent-certs\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-var-lib-kubelet\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f2f96e82-4aa4-48c0-8386-d629ce669284-lib-modules\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/70523d1b-ee7e-495d-a301-ade04b748746-multus-daemon-config\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-etc-kubernetes\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006676 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7ss\" (UniqueName: \"kubernetes.io/projected/70523d1b-ee7e-495d-a301-ade04b748746-kube-api-access-fj7ss\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-tmp\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.008699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.006714 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/70523d1b-ee7e-495d-a301-ade04b748746-etc-kubernetes\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.009694 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.009259 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-etc-tuned\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.009694 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.009432 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e0ea746-b225-44d5-9981-db1ecfc29ca5-agent-certs\") pod \"konnectivity-agent-xmdzd\" (UID: \"0e0ea746-b225-44d5-9981-db1ecfc29ca5\") " pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:53.009694 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.009614 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f2f96e82-4aa4-48c0-8386-d629ce669284-tmp\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.012832 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.012796 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:53.012972 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.012838 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:53.012972 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.012855 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:53.012972 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.012924 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:36:53.512907026 +0000 UTC m=+3.188968795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:53.015504 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.015479 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwj8\" (UniqueName: \"kubernetes.io/projected/f2f96e82-4aa4-48c0-8386-d629ce669284-kube-api-access-5kwj8\") pod \"tuned-6nm5r\" (UID: \"f2f96e82-4aa4-48c0-8386-d629ce669284\") " pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.016217 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.016195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ckp\" (UniqueName: \"kubernetes.io/projected/cadf1967-b262-4bdd-a933-968447f88cf1-kube-api-access-s4ckp\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:53.016309 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.016288 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7ss\" (UniqueName: \"kubernetes.io/projected/70523d1b-ee7e-495d-a301-ade04b748746-kube-api-access-fj7ss\") pod \"multus-zr4fb\" (UID: \"70523d1b-ee7e-495d-a301-ade04b748746\") " pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.041585 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.041508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" event={"ID":"44bd8c551779de4c442e1d9ae5fc2be1","Type":"ContainerStarted","Data":"627b59866197b21798163f99029013c636ccbd0e5d8e125a3a2af556c6c44c85"} Jul 10 16:36:53.042508 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.042488 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" event={"ID":"f4b29e5d3cfbcc3884d91f6b4f3cf71c","Type":"ContainerStarted","Data":"bc0940807454b884c52444365723dc5867af7474dc5842bd2c8111e059379b7d"} Jul 10 16:36:53.093162 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.093123 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r276w" Jul 10 16:36:53.102455 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.102424 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlpg5" Jul 10 16:36:53.103009 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.102979 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec6e9897_b9bd_4f00_acbd_72aea3e101e7.slice/crio-8f08a202bf66716da4a566c01d773226c6330124d07b097ff18752572815eed7 WatchSource:0}: Error finding container 8f08a202bf66716da4a566c01d773226c6330124d07b097ff18752572815eed7: Status 404 returned error can't find the container with id 8f08a202bf66716da4a566c01d773226c6330124d07b097ff18752572815eed7 Jul 10 16:36:53.111303 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.111280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:36:53.117293 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.117270 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" Jul 10 16:36:53.119486 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.119460 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bdf848b_cc24_4998_bb53_4a3ea919e2f6.slice/crio-6431ae8e7edd03668b10ed0fd71a21a6198d5044f253d09c30d7083e1123cde1 WatchSource:0}: Error finding container 6431ae8e7edd03668b10ed0fd71a21a6198d5044f253d09c30d7083e1123cde1: Status 404 returned error can't find the container with id 6431ae8e7edd03668b10ed0fd71a21a6198d5044f253d09c30d7083e1123cde1 Jul 10 16:36:53.124584 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.124562 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-2ndkl" Jul 10 16:36:53.127426 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.127403 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod489aa976_73ae_42ea_9c9c_c2490cab6607.slice/crio-4996302839d2d9dbdc06e8b49171eb66aa7c3a66f60409b358046d01c5c865c6 WatchSource:0}: Error finding container 4996302839d2d9dbdc06e8b49171eb66aa7c3a66f60409b358046d01c5c865c6: Status 404 returned error can't find the container with id 4996302839d2d9dbdc06e8b49171eb66aa7c3a66f60409b358046d01c5c865c6 Jul 10 16:36:53.131231 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.131211 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-692mv" Jul 10 16:36:53.133223 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.133146 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff236cfe_018a_42db_a189_3b1d84b276cf.slice/crio-643d1db84543bfa50c6e29b3abe756ba743f127d29888e35d6e0683b85a14a60 WatchSource:0}: Error finding container 643d1db84543bfa50c6e29b3abe756ba743f127d29888e35d6e0683b85a14a60: Status 404 returned error can't find the container with id 643d1db84543bfa50c6e29b3abe756ba743f127d29888e35d6e0683b85a14a60 Jul 10 16:36:53.138903 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.138884 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" Jul 10 16:36:53.139141 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.139123 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3758299_2838_43fd_ad6a_bc8b885579cd.slice/crio-4e9fa124772e0d31811d69e17f4f4a3821435d0fdc15bbe25c9853a96a1f71e3 WatchSource:0}: Error finding container 4e9fa124772e0d31811d69e17f4f4a3821435d0fdc15bbe25c9853a96a1f71e3: Status 404 returned error can't find the container with id 4e9fa124772e0d31811d69e17f4f4a3821435d0fdc15bbe25c9853a96a1f71e3 Jul 10 16:36:53.145407 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.145386 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zr4fb" Jul 10 16:36:53.147473 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.147323 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f96e82_4aa4_48c0_8386_d629ce669284.slice/crio-e350ce673df0829a3632a1e1948f303801e4bec82602d568ac709a6d839022d1 WatchSource:0}: Error finding container e350ce673df0829a3632a1e1948f303801e4bec82602d568ac709a6d839022d1: Status 404 returned error can't find the container with id e350ce673df0829a3632a1e1948f303801e4bec82602d568ac709a6d839022d1 Jul 10 16:36:53.151011 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.150782 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:36:53.155542 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.155489 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70523d1b_ee7e_495d_a301_ade04b748746.slice/crio-647c3b7786cfeedc3409348a0ecb884553eb4a2276ba0e0bf480892b56195761 WatchSource:0}: Error finding container 647c3b7786cfeedc3409348a0ecb884553eb4a2276ba0e0bf480892b56195761: Status 404 returned error can't find the container with id 647c3b7786cfeedc3409348a0ecb884553eb4a2276ba0e0bf480892b56195761 Jul 10 16:36:53.161327 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:36:53.161304 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e0ea746_b225_44d5_9981_db1ecfc29ca5.slice/crio-65a04b1a6c1a4b923d91cf9023f8ab581c30c1a4bdeb29d3851e205ec9f860de WatchSource:0}: Error finding container 65a04b1a6c1a4b923d91cf9023f8ab581c30c1a4bdeb29d3851e205ec9f860de: Status 404 returned error can't find the container with id 65a04b1a6c1a4b923d91cf9023f8ab581c30c1a4bdeb29d3851e205ec9f860de Jul 10 16:36:53.221545 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.221514 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:53.510227 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.510122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:53.510227 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.510197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:53.510449 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.510288 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:53.510449 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.510327 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:53.510449 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.510375 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.510352683 +0000 UTC m=+4.186414461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:53.510449 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.510396 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.510386108 +0000 UTC m=+4.186447872 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:53.611415 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.611370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:53.611602 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.611535 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:53.611602 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.611559 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:53.611602 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.611571 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:53.611803 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:53.611638 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:36:54.611618747 +0000 UTC m=+4.287680514 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:53.832779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.832543 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:31:52 +0000 UTC" deadline="2028-01-23 05:20:05.812725212 +0000 UTC" Jul 10 16:36:53.833288 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:53.832801 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13476h43m11.979948156s" Jul 10 16:36:54.037755 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.037621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:54.038021 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.037839 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:36:54.048589 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.048493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlpg5" event={"ID":"76d47f55-7101-4301-add0-569f73b6e8e7","Type":"ContainerStarted","Data":"b56eaeb5cd5d8a3af67086f4608edb1fad3cc1e004b2251b05db00db73378204"} Jul 10 16:36:54.053646 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.053546 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-692mv" event={"ID":"a3758299-2838-43fd-ad6a-bc8b885579cd","Type":"ContainerStarted","Data":"4e9fa124772e0d31811d69e17f4f4a3821435d0fdc15bbe25c9853a96a1f71e3"} Jul 10 16:36:54.071917 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.070627 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ndkl" event={"ID":"ff236cfe-018a-42db-a189-3b1d84b276cf","Type":"ContainerStarted","Data":"643d1db84543bfa50c6e29b3abe756ba743f127d29888e35d6e0683b85a14a60"} Jul 10 16:36:54.089259 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.089222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerStarted","Data":"8f08a202bf66716da4a566c01d773226c6330124d07b097ff18752572815eed7"} Jul 10 16:36:54.098550 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.098515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmdzd" event={"ID":"0e0ea746-b225-44d5-9981-db1ecfc29ca5","Type":"ContainerStarted","Data":"65a04b1a6c1a4b923d91cf9023f8ab581c30c1a4bdeb29d3851e205ec9f860de"} Jul 10 16:36:54.100650 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.100621 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zr4fb" event={"ID":"70523d1b-ee7e-495d-a301-ade04b748746","Type":"ContainerStarted","Data":"647c3b7786cfeedc3409348a0ecb884553eb4a2276ba0e0bf480892b56195761"} Jul 10 16:36:54.103513 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.103484 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" event={"ID":"f2f96e82-4aa4-48c0-8386-d629ce669284","Type":"ContainerStarted","Data":"e350ce673df0829a3632a1e1948f303801e4bec82602d568ac709a6d839022d1"} Jul 10 16:36:54.110633 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.110493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" event={"ID":"489aa976-73ae-42ea-9c9c-c2490cab6607","Type":"ContainerStarted","Data":"4996302839d2d9dbdc06e8b49171eb66aa7c3a66f60409b358046d01c5c865c6"} Jul 10 16:36:54.116883 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.116853 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"6431ae8e7edd03668b10ed0fd71a21a6198d5044f253d09c30d7083e1123cde1"} Jul 10 16:36:54.416247 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.416149 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 10 16:36:54.518441 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.518397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:54.518623 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.518467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:54.518700 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.518657 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:54.518754 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.518724 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:56.518704481 +0000 UTC m=+6.194766261 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:54.519261 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.519238 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:54.519357 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.519308 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:36:56.519289539 +0000 UTC m=+6.195351306 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:54.619584 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.619541 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:54.619876 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.619853 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:54.619971 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.619886 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:54.619971 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.619900 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:54.620084 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:54.619976 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:36:56.619945679 +0000 UTC m=+6.296007467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:54.834049 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.833919 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-09 16:31:52 +0000 UTC" deadline="2028-04-08 17:15:22.31881853 +0000 UTC" Jul 10 16:36:54.834049 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:54.833984 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15312h38m27.484838931s" Jul 10 16:36:55.041229 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:55.040455 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:55.041229 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:55.040597 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:36:55.041229 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:55.041000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:55.041229 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:55.041102 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:36:55.135210 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:55.135124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" event={"ID":"f4b29e5d3cfbcc3884d91f6b4f3cf71c","Type":"ContainerStarted","Data":"6d4bfbb6e9f4b7ec760936f241e6f3896b05d75d6a9e613902b50d28fef914bf"} Jul 10 16:36:56.038077 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:56.037584 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:56.038077 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.037713 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:56.537655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:56.537714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.537886 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.537949 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:00.537931251 +0000 UTC m=+10.213993020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.538337 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:56.538429 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.538388 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:00.538371078 +0000 UTC m=+10.214432864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:36:56.639757 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:56.639116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:56.639757 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.639311 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:36:56.639757 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.639330 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:36:56.639757 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.639343 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:56.639757 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:56.639405 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:37:00.639386924 +0000 UTC m=+10.315448689 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:36:57.037972 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:57.037872 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:57.037972 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:57.037894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:57.038509 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:57.038053 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:36:57.038570 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:57.038502 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:36:58.037769 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:58.037368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:36:58.037769 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:58.037540 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:36:58.151855 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:58.151816 2568 generic.go:356] "Generic (PLEG): container finished" podID="f4b29e5d3cfbcc3884d91f6b4f3cf71c" containerID="6d4bfbb6e9f4b7ec760936f241e6f3896b05d75d6a9e613902b50d28fef914bf" exitCode=0 Jul 10 16:36:58.152302 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:58.151876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" event={"ID":"f4b29e5d3cfbcc3884d91f6b4f3cf71c","Type":"ContainerDied","Data":"6d4bfbb6e9f4b7ec760936f241e6f3896b05d75d6a9e613902b50d28fef914bf"} Jul 10 16:36:59.037557 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:59.037522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:36:59.037795 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:59.037680 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:36:59.037795 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:36:59.037708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:36:59.037936 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:36:59.037834 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:00.038385 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:00.038350 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:00.038889 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.038506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:00.573030 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:00.572994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:00.573227 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:00.573050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:00.573227 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.573169 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:00.573227 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.573176 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:00.573227 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.573220 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:08.573207844 +0000 UTC m=+18.249269607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:00.573434 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.573236 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:08.57322921 +0000 UTC m=+18.249290972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:00.673779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:00.673724 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:00.673929 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.673883 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:37:00.673929 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.673901 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:37:00.673929 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.673913 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:00.674082 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:00.673979 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:37:08.673960011 +0000 UTC m=+18.350021782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:01.038896 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:01.038545 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:01.038896 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:01.038660 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:01.038896 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:01.038730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:01.038896 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:01.038847 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:02.037635 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:02.037602 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:02.037845 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:02.037726 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:03.038147 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:03.038112 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:03.038595 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:03.038240 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:03.038595 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:03.038292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:03.038595 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:03.038409 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:04.037551 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:04.037292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:04.037551 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:04.037429 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:05.037918 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:05.037842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:05.037918 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:05.037880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:05.038347 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:05.037949 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:05.038347 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:05.038071 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:06.037814 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:06.037781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:06.037970 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:06.037908 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:07.037879 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:07.037838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:07.038072 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:07.037846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:07.038072 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:07.037974 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:07.038423 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:07.038090 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:08.037753 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:08.037704 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:08.037989 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.037851 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:08.632406 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:08.632368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:08.632950 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:08.632419 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:08.632950 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.632533 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:08.632950 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.632543 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:08.632950 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.632602 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:24.632581846 +0000 UTC m=+34.308643636 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:08.632950 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.632622 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:24.632611181 +0000 UTC m=+34.308672944 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:08.733281 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:08.733243 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:08.733457 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.733422 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:37:08.733457 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.733445 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:37:08.733457 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.733457 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:08.733596 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:08.733522 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:37:24.733502505 +0000 UTC m=+34.409564283 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:09.038200 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:09.038096 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:09.038380 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:09.038230 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:09.038380 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:09.038280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:09.038490 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:09.038388 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:10.037644 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:10.037607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:10.038114 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:10.037717 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:11.039082 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:11.039041 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:11.039519 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:11.039157 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:11.039519 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:11.039214 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:11.039519 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:11.039353 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:12.037909 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:12.037870 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:12.038083 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:12.037992 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:13.038238 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.037989 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:13.039238 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.038055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:13.039238 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:13.038387 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:13.039238 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:13.038468 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:13.183341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.183304 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" event={"ID":"f4b29e5d3cfbcc3884d91f6b4f3cf71c","Type":"ContainerStarted","Data":"b49509985b39c4b60e406ff2a596d81846491432abd0c9087de451dc179608b6"} Jul 10 16:37:13.184541 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.184503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xmdzd" event={"ID":"0e0ea746-b225-44d5-9981-db1ecfc29ca5","Type":"ContainerStarted","Data":"a1c2a91183d461707dc9184938d8facf918a0b34fcb7bd9c7d089850c6edbb7e"} Jul 10 16:37:13.185786 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.185764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zr4fb" event={"ID":"70523d1b-ee7e-495d-a301-ade04b748746","Type":"ContainerStarted","Data":"22f5dd2b96ab832de8f998be9c76a782319dd7e44386bb124b5d079ef91ea849"} Jul 10 16:37:13.187127 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.187098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" event={"ID":"f2f96e82-4aa4-48c0-8386-d629ce669284","Type":"ContainerStarted","Data":"beeff83fd8e5854d28020a9a6625aec52d25e0f5874636b18e00969faf16d424"} Jul 10 16:37:13.188351 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.188327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" event={"ID":"489aa976-73ae-42ea-9c9c-c2490cab6607","Type":"ContainerStarted","Data":"2eac28c0b31abbd4e97bee14931fb518d677fa5186c8ab6a317376906742989c"} Jul 10 16:37:13.189807 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.189791 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:13.190127 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.190107 2568 generic.go:356] "Generic (PLEG): container finished" podID="9bdf848b-cc24-4998-bb53-4a3ea919e2f6" containerID="2218a1803221765ff0ed8459018b652d89c63607fb7c7eb7b876cb7574eaf54f" exitCode=1 Jul 10 16:37:13.190204 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.190173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerDied","Data":"2218a1803221765ff0ed8459018b652d89c63607fb7c7eb7b876cb7574eaf54f"} Jul 10 16:37:13.190273 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.190255 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"cb3491572a19ec5c7a5c1ef84479dc3012254706b6553a363ec7616c3b95e965"} Jul 10 16:37:13.191967 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.191942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" event={"ID":"44bd8c551779de4c442e1d9ae5fc2be1","Type":"ContainerStarted","Data":"10885e453f912be1aae32729012d9fa77f0c82b7d56477d3de1d8024a614763f"} Jul 10 16:37:13.193395 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.193373 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-692mv" event={"ID":"a3758299-2838-43fd-ad6a-bc8b885579cd","Type":"ContainerStarted","Data":"e8428471555fec9b4ab0e8afbbe12c68d604e3b8d7f622d3d156123059a8796b"} Jul 10 16:37:13.194934 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.194902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-2ndkl" event={"ID":"ff236cfe-018a-42db-a189-3b1d84b276cf","Type":"ContainerStarted","Data":"dff72fb834960d37d5837cd5880f0558d61385a7f6f612081d9ce592f89b61b4"} Jul 10 16:37:13.196613 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.196590 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="67564df773221a3e72e6718a65b33c924af682d8689bd99b0da00383fcf442ae" exitCode=0 Jul 10 16:37:13.196700 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.196622 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"67564df773221a3e72e6718a65b33c924af682d8689bd99b0da00383fcf442ae"} Jul 10 16:37:13.205700 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.205662 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xmdzd" podStartSLOduration=2.593722762 podStartE2EDuration="22.205651s" totalImagesPullingTime="19.611928238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.205473575 +0000 UTC m=+22.881535358" watchObservedRunningTime="2026-07-10 16:37:13.205651 +0000 UTC m=+22.881712783" Jul 10 16:37:13.218230 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.218196 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-107.ec2.internal" podStartSLOduration=21.218184504 podStartE2EDuration="21.218184504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.217930067 +0000 UTC m=+22.893991851" watchObservedRunningTime="2026-07-10 16:37:13.218184504 +0000 UTC m=+22.894246286" Jul 10 16:37:13.245393 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.245358 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-2ndkl" podStartSLOduration=2.5837303609999998 podStartE2EDuration="22.245345369s" totalImagesPullingTime="19.661615008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.245342957 +0000 UTC m=+22.921404741" watchObservedRunningTime="2026-07-10 16:37:13.245345369 +0000 UTC m=+22.921407153" Jul 10 16:37:13.245491 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.245448 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-107.ec2.internal" podStartSLOduration=21.245444042 podStartE2EDuration="21.245444042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.231720171 +0000 UTC m=+22.907781955" watchObservedRunningTime="2026-07-10 16:37:13.245444042 +0000 UTC m=+22.921505825" Jul 10 16:37:13.259188 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.259153 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-692mv" podStartSLOduration=6.505214018 podStartE2EDuration="22.259141688s" totalImagesPullingTime="15.75392767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.258554058 +0000 UTC m=+22.934615842" watchObservedRunningTime="2026-07-10 16:37:13.259141688 +0000 UTC m=+22.935203473" Jul 10 16:37:13.295795 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.295643 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zr4fb" podStartSLOduration=2.6053920440000002 podStartE2EDuration="22.295628234s" totalImagesPullingTime="19.69023619s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.295159171 +0000 UTC m=+22.971220955" watchObservedRunningTime="2026-07-10 16:37:13.295628234 +0000 UTC m=+22.971690018" Jul 10 16:37:13.295924 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:13.295851 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6nm5r" podStartSLOduration=2.669112777 podStartE2EDuration="22.295844387s" totalImagesPullingTime="19.62673161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:13.277696463 +0000 UTC m=+22.953758247" watchObservedRunningTime="2026-07-10 16:37:13.295844387 +0000 UTC m=+22.971906171" Jul 10 16:37:14.037923 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.037891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:14.038088 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:14.038047 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:14.204877 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.204847 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:14.205318 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.205277 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"02419aea380a09dea8db9b812fb592a2579d67132b917da66e97fcd61e415154"} Jul 10 16:37:14.205407 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.205387 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"30150df5e90b7fc70ae93d92da9f78677a2120c9f2927f572672b91c9ac0f162"} Jul 10 16:37:14.205529 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.205442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"b826c43d7292a68ec541702a54d0fe742650b08c76b8a67fedeee7f2c3e2454e"} Jul 10 16:37:14.205529 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.205476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"72f6558dee549dd08e14924e8a8805722552520eb7007e3951ee7e319f6725d1"} Jul 10 16:37:14.207407 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.207304 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlpg5" event={"ID":"76d47f55-7101-4301-add0-569f73b6e8e7","Type":"ContainerStarted","Data":"06abba80670944de7f5edbbe407974962c4338e0c2c49d3832be90c0968d8da3"} Jul 10 16:37:14.224382 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.224329 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tlpg5" podStartSLOduration=3.551832398 podStartE2EDuration="23.224316343s" totalImagesPullingTime="19.672483945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:14.223717086 +0000 UTC m=+23.899778871" watchObservedRunningTime="2026-07-10 16:37:14.224316343 +0000 UTC m=+23.900378127" Jul 10 16:37:14.404441 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.404351 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:37:14.404968 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.404949 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:37:14.708445 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.708423 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 10 16:37:14.860583 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.860478 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-10T16:37:14.708438613Z","UUID":"afe2aa61-3a04-4afd-9bdd-35aeb4625626","Handler":null,"Name":"","Endpoint":""} Jul 10 16:37:14.862560 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.862538 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 10 16:37:14.862682 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:14.862581 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 10 16:37:15.040922 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:15.040855 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:15.040922 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:15.040885 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:15.041093 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:15.040967 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:15.041093 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:15.041074 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:15.212049 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:15.212011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" event={"ID":"489aa976-73ae-42ea-9c9c-c2490cab6607","Type":"ContainerStarted","Data":"f912a1e92766e1567ff02caaa4b71824fcac6ee83b10fc51291f104dc1cd4416"} Jul 10 16:37:16.037847 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.037640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:16.038028 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:16.037950 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:16.215629 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.215597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" event={"ID":"489aa976-73ae-42ea-9c9c-c2490cab6607","Type":"ContainerStarted","Data":"600ea911af7c848e680bfc035bcc800af7bd7da118cf21c72c418fae66e805c3"} Jul 10 16:37:16.218555 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.218537 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:16.218915 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.218903 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 16:37:16.218993 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.218916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"6577cc31f08639ddf0dabab6880de30b7487f7036624f31638e0fb1210431599"} Jul 10 16:37:16.235442 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:16.235398 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tbszv" podStartSLOduration=2.712529587 podStartE2EDuration="25.235382257s" totalImagesPullingTime="22.52285267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:16.23481639 +0000 UTC m=+25.910878176" watchObservedRunningTime="2026-07-10 16:37:16.235382257 +0000 UTC m=+25.911444040" Jul 10 16:37:17.037940 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:17.037904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:17.038186 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:17.037904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:17.038186 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:17.038055 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:17.038186 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:17.038123 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:18.037735 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:18.037662 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:18.038315 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:18.037792 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:18.226322 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:18.226292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:19.037901 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.037710 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:19.038649 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.037753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:19.038649 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:19.037985 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:19.038649 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:19.038045 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:19.231553 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.231526 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:19.231947 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.231923 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"e0b29f23440bc8c1405e36423b866a59c7cc0bc128fb80757bef81818bcb873c"} Jul 10 16:37:19.232275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.232249 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:19.232389 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.232355 2568 scope.go:117] "RemoveContainer" containerID="2218a1803221765ff0ed8459018b652d89c63607fb7c7eb7b876cb7574eaf54f" Jul 10 16:37:19.234026 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.233998 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="594c34fec223e608b486cb2cebb6b262f7adce6f7c889d805fcbf546b7c88083" exitCode=0 Jul 10 16:37:19.234109 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.234039 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"594c34fec223e608b486cb2cebb6b262f7adce6f7c889d805fcbf546b7c88083"} Jul 10 16:37:19.248114 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:19.248096 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:20.037478 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.037441 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:20.037688 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:20.037552 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:20.239658 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.239635 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:37:20.240058 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.240033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" event={"ID":"9bdf848b-cc24-4998-bb53-4a3ea919e2f6","Type":"ContainerStarted","Data":"b80dce5d45c712c654db3542e0b2282b13f7c023d1075c706c1b7bb78687c36a"} Jul 10 16:37:20.243134 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.242465 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:20.243134 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.242530 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:20.261090 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.260946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:20.273139 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.273084 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" podStartSLOduration=9.442115257 podStartE2EDuration="29.273063449s" totalImagesPullingTime="19.830948192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:20.272836723 +0000 UTC m=+29.948898507" watchObservedRunningTime="2026-07-10 16:37:20.273063449 +0000 UTC m=+29.949125233" Jul 10 16:37:20.439131 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.439078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-59mt6"] Jul 10 16:37:20.439286 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.439233 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:20.439372 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:20.439344 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:20.439643 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.439624 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhcft"] Jul 10 16:37:20.439703 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.439692 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:20.439788 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:20.439773 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:20.441178 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.441154 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mssf"] Jul 10 16:37:20.441269 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:20.441256 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:20.441362 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:20.441343 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:21.244394 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:21.244361 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="9230b6b874218db2c985aa3b8fd03dabbe119f2c9e477c68f2870750684e4f78" exitCode=0 Jul 10 16:37:21.245008 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:21.244449 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"9230b6b874218db2c985aa3b8fd03dabbe119f2c9e477c68f2870750684e4f78"} Jul 10 16:37:22.038293 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.038220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:22.038293 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.038234 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:22.038531 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.038321 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:22.038531 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:22.038321 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:22.038531 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:22.038404 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:22.038531 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:22.038485 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:22.530485 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.530454 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:37:22.531164 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.530574 2568 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 10 16:37:22.531253 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:22.531239 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xmdzd" Jul 10 16:37:23.252413 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:23.252376 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="21b3fb168a6f6809684fb1b715923a35444c132d22f9a634e4dfd0ccb5cfc14c" exitCode=0 Jul 10 16:37:23.252586 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:23.252460 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"21b3fb168a6f6809684fb1b715923a35444c132d22f9a634e4dfd0ccb5cfc14c"} Jul 10 16:37:24.038298 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.038266 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:24.038696 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.038367 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:24.038696 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.038273 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:24.038696 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.038397 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:24.038696 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.038480 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:24.038696 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.038550 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:24.654501 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.654462 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:24.654658 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.654516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:24.654658 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.654620 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:24.654658 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.654631 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:24.654843 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.654676 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret podName:29764102-4050-48c5-bc52-cc0e03dfc516 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:56.654658899 +0000 UTC m=+66.330720675 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret") pod "global-pull-secret-syncer-nhcft" (UID: "29764102-4050-48c5-bc52-cc0e03dfc516") : object "kube-system"/"original-pull-secret" not registered Jul 10 16:37:24.654843 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.654694 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:56.654686331 +0000 UTC m=+66.330748093 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 10 16:37:24.755666 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:24.755586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:24.755821 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.755774 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 10 16:37:24.755821 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.755798 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 10 16:37:24.755821 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.755808 2568 projected.go:196] Error preparing data for projected volume kube-api-access-xj7st for pod openshift-network-diagnostics/network-check-target-59mt6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:24.755941 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:24.755860 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st podName:c28d9e8f-fa3d-421a-8e64-2c896f9b029f nodeName:}" failed. No retries permitted until 2026-07-10 16:37:56.755845141 +0000 UTC m=+66.431906907 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xj7st" (UniqueName: "kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st") pod "network-check-target-59mt6" (UID: "c28d9e8f-fa3d-421a-8e64-2c896f9b029f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 10 16:37:26.038275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.038236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:26.038275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.038275 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:26.038796 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.038277 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:26.038796 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:26.038374 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-59mt6" podUID="c28d9e8f-fa3d-421a-8e64-2c896f9b029f" Jul 10 16:37:26.038796 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:26.038469 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nhcft" podUID="29764102-4050-48c5-bc52-cc0e03dfc516" Jul 10 16:37:26.038796 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:26.038579 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:37:26.596770 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.596520 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-107.ec2.internal" event="NodeReady" Jul 10 16:37:26.596927 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.596844 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 10 16:37:26.640926 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.640896 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8cdld"] Jul 10 16:37:26.642806 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.642776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.644141 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.644121 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2wq72"] Jul 10 16:37:26.645855 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.645698 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 10 16:37:26.645855 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.645753 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 10 16:37:26.645855 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.645769 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqrx8\"" Jul 10 16:37:26.646074 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.646030 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.648493 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.648468 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-xrt5s\"" Jul 10 16:37:26.649280 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.649258 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 10 16:37:26.649280 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.649275 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 10 16:37:26.649419 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.649275 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 10 16:37:26.657013 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.656960 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8cdld"] Jul 10 16:37:26.657768 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.657729 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2wq72"] Jul 10 16:37:26.770641 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397527c-1fa8-415d-ac0e-61dadab748f1-tmp-dir\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.770858 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b397527c-1fa8-415d-ac0e-61dadab748f1-config-volume\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.770858 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74daaba-34e5-48e5-92b0-70066cce4536-cert\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.770858 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770722 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqdh\" (UniqueName: \"kubernetes.io/projected/a74daaba-34e5-48e5-92b0-70066cce4536-kube-api-access-2tqdh\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.770858 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770825 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.771050 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.770871 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gpzd\" (UniqueName: \"kubernetes.io/projected/b397527c-1fa8-415d-ac0e-61dadab748f1-kube-api-access-2gpzd\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872301 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b397527c-1fa8-415d-ac0e-61dadab748f1-config-volume\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872301 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74daaba-34e5-48e5-92b0-70066cce4536-cert\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.872301 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872259 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqdh\" (UniqueName: \"kubernetes.io/projected/a74daaba-34e5-48e5-92b0-70066cce4536-kube-api-access-2tqdh\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.872301 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872299 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872606 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gpzd\" (UniqueName: \"kubernetes.io/projected/b397527c-1fa8-415d-ac0e-61dadab748f1-kube-api-access-2gpzd\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872606 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397527c-1fa8-415d-ac0e-61dadab748f1-tmp-dir\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872699 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872658 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b397527c-1fa8-415d-ac0e-61dadab748f1-tmp-dir\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.872804 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:26.872776 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:26.872927 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:26.872849 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:27.372828627 +0000 UTC m=+37.048890400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:26.872927 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.872896 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b397527c-1fa8-415d-ac0e-61dadab748f1-config-volume\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.877293 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.877267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a74daaba-34e5-48e5-92b0-70066cce4536-cert\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.881436 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.881377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gpzd\" (UniqueName: \"kubernetes.io/projected/b397527c-1fa8-415d-ac0e-61dadab748f1-kube-api-access-2gpzd\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:26.881657 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.881636 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqdh\" (UniqueName: \"kubernetes.io/projected/a74daaba-34e5-48e5-92b0-70066cce4536-kube-api-access-2tqdh\") pod \"ingress-canary-2wq72\" (UID: \"a74daaba-34e5-48e5-92b0-70066cce4536\") " pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:26.962596 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:26.962562 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2wq72" Jul 10 16:37:27.375822 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:27.375781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:27.376270 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:27.375951 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:27.376270 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:27.376025 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:28.376003433 +0000 UTC m=+38.052065217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:28.038012 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.037979 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:28.038198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.037979 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:28.038198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.037979 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:28.041209 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.041086 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:37:28.041209 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.041110 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:37:28.041209 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.041132 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:37:28.042341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.042313 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:37:28.042444 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.042350 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggg2b\"" Jul 10 16:37:28.042444 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.042383 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bc5mm\"" Jul 10 16:37:28.383442 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.383344 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:28.383952 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:28.383484 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:28.383952 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:28.383531 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:30.383518001 +0000 UTC m=+40.059579766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:28.445770 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:28.445729 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2wq72"] Jul 10 16:37:28.451533 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:37:28.451271 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74daaba_34e5_48e5_92b0_70066cce4536.slice/crio-8a0009dedd4a32b2f113292a252c77d382649fe1d9b08821a2745685460daf87 WatchSource:0}: Error finding container 8a0009dedd4a32b2f113292a252c77d382649fe1d9b08821a2745685460daf87: Status 404 returned error can't find the container with id 8a0009dedd4a32b2f113292a252c77d382649fe1d9b08821a2745685460daf87 Jul 10 16:37:29.267198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:29.267162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2wq72" event={"ID":"a74daaba-34e5-48e5-92b0-70066cce4536","Type":"ContainerStarted","Data":"8a0009dedd4a32b2f113292a252c77d382649fe1d9b08821a2745685460daf87"} Jul 10 16:37:29.269808 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:29.269780 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="56daea7b7b6e3d46d529b1a017c8c87db0d62430bb7284776f485452bd65e824" exitCode=0 Jul 10 16:37:29.269922 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:29.269827 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"56daea7b7b6e3d46d529b1a017c8c87db0d62430bb7284776f485452bd65e824"} Jul 10 16:37:30.276315 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:30.276277 2568 generic.go:356] "Generic (PLEG): container finished" podID="ec6e9897-b9bd-4f00-acbd-72aea3e101e7" containerID="2adcbc3ea0a585076b38a6c6533b0a52935b3db207a4f95fcf7752a4302fe48c" exitCode=0 Jul 10 16:37:30.276937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:30.276336 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerDied","Data":"2adcbc3ea0a585076b38a6c6533b0a52935b3db207a4f95fcf7752a4302fe48c"} Jul 10 16:37:30.397532 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:30.397490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:30.397718 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:30.397692 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:30.397815 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:30.397782 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:34.397761159 +0000 UTC m=+44.073822927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:31.282814 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:31.282774 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r276w" event={"ID":"ec6e9897-b9bd-4f00-acbd-72aea3e101e7","Type":"ContainerStarted","Data":"414f4763172fc2ee22f2f2ca6ecb43602651e77c6983eae1a23e0e9c3c70c854"} Jul 10 16:37:31.309103 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:31.308699 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r276w" podStartSLOduration=8.37000405 podStartE2EDuration="40.308675907s" totalImagesPullingTime="31.938671857s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:31.307104075 +0000 UTC m=+40.983165851" watchObservedRunningTime="2026-07-10 16:37:31.308675907 +0000 UTC m=+40.984737695" Jul 10 16:37:32.287133 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:32.287090 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2wq72" event={"ID":"a74daaba-34e5-48e5-92b0-70066cce4536","Type":"ContainerStarted","Data":"e1a8bcd4dae3384c8650383298dfb94bf613eab942db0ee387253762c7a7bd5e"} Jul 10 16:37:32.304288 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:32.304234 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2wq72" podStartSLOduration=2.9418007409999998 podStartE2EDuration="6.304218721s" totalImagesPullingTime="3.36241798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:37:32.30237402 +0000 UTC m=+41.978435801" watchObservedRunningTime="2026-07-10 16:37:32.304218721 +0000 UTC m=+41.980280511" Jul 10 16:37:34.426292 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:34.426251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:34.426645 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:34.426384 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:34.426645 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:34.426442 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:42.42642597 +0000 UTC m=+52.102487737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:42.482544 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:42.482508 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:42.483026 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:42.482682 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:42.483026 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:42.482766 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:37:58.482735056 +0000 UTC m=+68.158796818 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:37:52.264897 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:52.264869 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-scsl6" Jul 10 16:37:56.683832 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.683778 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:37:56.683832 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.683842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:56.686729 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.686707 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 10 16:37:56.687012 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.686995 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 10 16:37:56.695007 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:56.694983 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 16:37:56.695097 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:56.695044 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:39:00.695028325 +0000 UTC m=+130.371090087 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : secret "metrics-daemon-secret" not found Jul 10 16:37:56.698051 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.698030 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/29764102-4050-48c5-bc52-cc0e03dfc516-original-pull-secret\") pod \"global-pull-secret-syncer-nhcft\" (UID: \"29764102-4050-48c5-bc52-cc0e03dfc516\") " pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:56.784734 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.784684 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:56.787809 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.787791 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 10 16:37:56.797396 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.797376 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 10 16:37:56.807786 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.807758 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7st\" (UniqueName: \"kubernetes.io/projected/c28d9e8f-fa3d-421a-8e64-2c896f9b029f-kube-api-access-xj7st\") pod \"network-check-target-59mt6\" (UID: \"c28d9e8f-fa3d-421a-8e64-2c896f9b029f\") " pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:56.856787 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.856732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nhcft" Jul 10 16:37:56.865298 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.865273 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-ggg2b\"" Jul 10 16:37:56.872887 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.872863 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:37:56.999701 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:56.999675 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nhcft"] Jul 10 16:37:57.002811 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:37:57.002786 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29764102_4050_48c5_bc52_cc0e03dfc516.slice/crio-1a0616bc1ba6415e6089ae7df8e68a0b183ca8406ebcd6edc767b6ad7fb8f255 WatchSource:0}: Error finding container 1a0616bc1ba6415e6089ae7df8e68a0b183ca8406ebcd6edc767b6ad7fb8f255: Status 404 returned error can't find the container with id 1a0616bc1ba6415e6089ae7df8e68a0b183ca8406ebcd6edc767b6ad7fb8f255 Jul 10 16:37:57.010782 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:57.010759 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-59mt6"] Jul 10 16:37:57.013542 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:37:57.013520 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc28d9e8f_fa3d_421a_8e64_2c896f9b029f.slice/crio-bbf4e76a041ecf212b7d283c710a8c5fbcee428e0bb4ab7715c11f6b325935c2 WatchSource:0}: Error finding container bbf4e76a041ecf212b7d283c710a8c5fbcee428e0bb4ab7715c11f6b325935c2: Status 404 returned error can't find the container with id bbf4e76a041ecf212b7d283c710a8c5fbcee428e0bb4ab7715c11f6b325935c2 Jul 10 16:37:57.346556 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:57.346452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-59mt6" event={"ID":"c28d9e8f-fa3d-421a-8e64-2c896f9b029f","Type":"ContainerStarted","Data":"bbf4e76a041ecf212b7d283c710a8c5fbcee428e0bb4ab7715c11f6b325935c2"} Jul 10 16:37:57.347506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:57.347486 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhcft" event={"ID":"29764102-4050-48c5-bc52-cc0e03dfc516","Type":"ContainerStarted","Data":"1a0616bc1ba6415e6089ae7df8e68a0b183ca8406ebcd6edc767b6ad7fb8f255"} Jul 10 16:37:58.495063 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:37:58.495030 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:37:58.495482 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:58.495158 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:37:58.495482 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:37:58.495218 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:38:30.495201379 +0000 UTC m=+100.171263147 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:38:03.364723 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.364687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-59mt6" event={"ID":"c28d9e8f-fa3d-421a-8e64-2c896f9b029f","Type":"ContainerStarted","Data":"d3deb7f49f127d1c446b99d5cd6f8db3702c736a1b0dc146680ece28e07ad91e"} Jul 10 16:38:03.365222 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.364807 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:38:03.366156 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.366133 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nhcft" event={"ID":"29764102-4050-48c5-bc52-cc0e03dfc516","Type":"ContainerStarted","Data":"c7d8e6e36aad189220ba5319d649ecdb4efefe047df39cb5c146e0a6ad7cf93a"} Jul 10 16:38:03.381336 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.381294 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-59mt6" podStartSLOduration=66.696912718 podStartE2EDuration="1m12.381281402s" totalImagesPullingTime="5.684368684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:38:03.379596517 +0000 UTC m=+73.055658314" watchObservedRunningTime="2026-07-10 16:38:03.381281402 +0000 UTC m=+73.057343208" Jul 10 16:38:03.393787 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.393719 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nhcft" podStartSLOduration=66.694622094 podStartE2EDuration="1m12.393702381s" totalImagesPullingTime="5.699080287s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:38:03.392912888 +0000 UTC m=+73.068974673" watchObservedRunningTime="2026-07-10 16:38:03.393702381 +0000 UTC m=+73.069764165" Jul 10 16:38:03.965992 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.965956 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq"] Jul 10 16:38:03.967888 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.967867 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:03.971675 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.971647 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 10 16:38:03.971675 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.971659 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 10 16:38:03.971862 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.971659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 10 16:38:03.971862 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.971733 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 10 16:38:03.978779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:03.978755 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq"] Jul 10 16:38:04.015064 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.015030 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9"] Jul 10 16:38:04.020458 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.020432 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.023387 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.023364 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 10 16:38:04.023544 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.023460 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 10 16:38:04.023544 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.023476 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 10 16:38:04.023633 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.023577 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 10 16:38:04.029019 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.028998 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9"] Jul 10 16:38:04.030397 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.030377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-tmp\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.030506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.030433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-klusterlet-config\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.030506 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.030458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4j26\" (UniqueName: \"kubernetes.io/projected/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-kube-api-access-p4j26\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.130987 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.130950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abae4655-7375-433c-a9ae-3264b9a2336b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.130987 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.130991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8wp\" (UniqueName: \"kubernetes.io/projected/abae4655-7375-433c-a9ae-3264b9a2336b-kube-api-access-jw8wp\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131021 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-klusterlet-config\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4j26\" (UniqueName: \"kubernetes.io/projected/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-kube-api-access-p4j26\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-tmp\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.131237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.131526 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.131500 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-tmp\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.133595 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.133575 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-klusterlet-config\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.139651 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.139624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4j26\" (UniqueName: \"kubernetes.io/projected/b3ca4cb1-7f84-4bdc-83a5-e2931a44de42-kube-api-access-p4j26\") pod \"klusterlet-addon-workmgr-769558bdcf-ppcbq\" (UID: \"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.231579 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231494 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.231579 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.231579 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231554 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.231862 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abae4655-7375-433c-a9ae-3264b9a2336b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.231862 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8wp\" (UniqueName: \"kubernetes.io/projected/abae4655-7375-433c-a9ae-3264b9a2336b-kube-api-access-jw8wp\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.231862 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.231650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.232383 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.232355 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/abae4655-7375-433c-a9ae-3264b9a2336b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.234085 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.234060 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.234085 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.234073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.234224 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.234103 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-ca\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.234224 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.234149 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/abae4655-7375-433c-a9ae-3264b9a2336b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.239560 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.239536 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8wp\" (UniqueName: \"kubernetes.io/projected/abae4655-7375-433c-a9ae-3264b9a2336b-kube-api-access-jw8wp\") pod \"cluster-proxy-proxy-agent-7f784cd56-5cbn9\" (UID: \"abae4655-7375-433c-a9ae-3264b9a2336b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.277550 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.277504 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:04.341112 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.341086 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:38:04.401496 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.401356 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq"] Jul 10 16:38:04.404191 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:38:04.404165 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ca4cb1_7f84_4bdc_83a5_e2931a44de42.slice/crio-0e02591921e06296aa330f804fcdc6e8ca4c3f1a129b0287edb13ca59dcef9a0 WatchSource:0}: Error finding container 0e02591921e06296aa330f804fcdc6e8ca4c3f1a129b0287edb13ca59dcef9a0: Status 404 returned error can't find the container with id 0e02591921e06296aa330f804fcdc6e8ca4c3f1a129b0287edb13ca59dcef9a0 Jul 10 16:38:04.463357 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:04.463332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9"] Jul 10 16:38:04.466227 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:38:04.466200 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabae4655_7375_433c_a9ae_3264b9a2336b.slice/crio-6e00a2fe6c755ba5d08695c1ad6cd8b9d0136c1b99492f4e776293f54cf72c57 WatchSource:0}: Error finding container 6e00a2fe6c755ba5d08695c1ad6cd8b9d0136c1b99492f4e776293f54cf72c57: Status 404 returned error can't find the container with id 6e00a2fe6c755ba5d08695c1ad6cd8b9d0136c1b99492f4e776293f54cf72c57 Jul 10 16:38:05.375195 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:05.375154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerStarted","Data":"6e00a2fe6c755ba5d08695c1ad6cd8b9d0136c1b99492f4e776293f54cf72c57"} Jul 10 16:38:05.376571 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:05.376538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" event={"ID":"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42","Type":"ContainerStarted","Data":"0e02591921e06296aa330f804fcdc6e8ca4c3f1a129b0287edb13ca59dcef9a0"} Jul 10 16:38:09.388865 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:09.388828 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" event={"ID":"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42","Type":"ContainerStarted","Data":"c5f5807d3b4eb831a3a6729caad0929db43016044b3160eb5f1765e261adbc5a"} Jul 10 16:38:09.389327 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:09.389039 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:09.390686 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:09.390666 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:38:09.405289 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:09.405241 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" podStartSLOduration=2.397613332 podStartE2EDuration="6.405227741s" totalImagesPullingTime="4.007614409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:38:09.404362012 +0000 UTC m=+79.080423796" watchObservedRunningTime="2026-07-10 16:38:09.405227741 +0000 UTC m=+79.081289524" Jul 10 16:38:13.401083 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:13.400998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerStarted","Data":"8877fc09c36fe0b4fd4fd3766ae37d8ccba270fc7ed65a256f307676fd57fbf1"} Jul 10 16:38:13.401083 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:13.401047 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerStarted","Data":"498e5c8c99222e2d5bca224eeb509ceb450b075c3c5e6a48122b6c94806a059a"} Jul 10 16:38:13.401083 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:13.401064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerStarted","Data":"e9a643fc0309eb96566cdd18b146e69f2030268d30b29e6b9d27586e0d07b273"} Jul 10 16:38:13.421330 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:13.421268 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" podStartSLOduration=1.852025313 podStartE2EDuration="10.421252937s" totalImagesPullingTime="8.569227624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:38:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:38:13.420809543 +0000 UTC m=+83.096871323" watchObservedRunningTime="2026-07-10 16:38:13.421252937 +0000 UTC m=+83.097314715" Jul 10 16:38:30.504861 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:30.504804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:38:30.505295 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:38:30.504986 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:38:30.505295 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:38:30.505061 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:39:34.505045088 +0000 UTC m=+164.181106864 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:38:34.372275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:38:34.372247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-59mt6" Jul 10 16:39:00.722785 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:39:00.722729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:39:00.723265 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:00.722873 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 10 16:39:00.723265 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:00.722951 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs podName:cadf1967-b262-4bdd-a933-968447f88cf1 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:02.722934206 +0000 UTC m=+252.398995969 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs") pod "network-metrics-daemon-8mssf" (UID: "cadf1967-b262-4bdd-a933-968447f88cf1") : secret "metrics-daemon-secret" not found Jul 10 16:39:29.655985 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:29.655927 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8cdld" podUID="b397527c-1fa8-415d-ac0e-61dadab748f1" Jul 10 16:39:30.615771 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:39:30.615727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cdld" Jul 10 16:39:31.049605 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:31.049569 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8mssf" podUID="cadf1967-b262-4bdd-a933-968447f88cf1" Jul 10 16:39:34.561394 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:39:34.561351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:39:34.561906 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:34.561523 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 10 16:39:34.561906 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:39:34.561624 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls podName:b397527c-1fa8-415d-ac0e-61dadab748f1 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:36.561601849 +0000 UTC m=+286.237663632 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls") pod "dns-default-8cdld" (UID: "b397527c-1fa8-415d-ac0e-61dadab748f1") : secret "dns-default-metrics-tls" not found Jul 10 16:39:44.038200 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:39:44.038105 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:40:08.723595 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:08.723560 2568 generic.go:356] "Generic (PLEG): container finished" podID="b3ca4cb1-7f84-4bdc-83a5-e2931a44de42" containerID="c5f5807d3b4eb831a3a6729caad0929db43016044b3160eb5f1765e261adbc5a" exitCode=1 Jul 10 16:40:08.723595 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:08.723598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" event={"ID":"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42","Type":"ContainerDied","Data":"c5f5807d3b4eb831a3a6729caad0929db43016044b3160eb5f1765e261adbc5a"} Jul 10 16:40:08.724119 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:08.723917 2568 scope.go:117] "RemoveContainer" containerID="c5f5807d3b4eb831a3a6729caad0929db43016044b3160eb5f1765e261adbc5a" Jul 10 16:40:09.389815 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:09.389781 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:40:09.727636 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:09.727604 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" event={"ID":"b3ca4cb1-7f84-4bdc-83a5-e2931a44de42","Type":"ContainerStarted","Data":"79ce1aa66f70e14582b3cdd6f3c8797d6f618be0f621662386a85a17f622e447"} Jul 10 16:40:09.728041 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:09.727810 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:40:09.728446 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:09.728429 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-769558bdcf-ppcbq" Jul 10 16:40:37.501245 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.501213 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m4qbg"] Jul 10 16:40:37.503224 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.503207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.506735 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.506704 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 10 16:40:37.507033 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.507012 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 10 16:40:37.507994 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.507975 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2jlt9\"" Jul 10 16:40:37.508148 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.508129 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 10 16:40:37.508230 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.508138 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 10 16:40:37.520871 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.520839 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4qbg"] Jul 10 16:40:37.601426 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.601395 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc49631f-2768-44ba-815b-3603f99bd9ea-data-volume\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.601426 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.601429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc49631f-2768-44ba-815b-3603f99bd9ea-crio-socket\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.601645 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.601447 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blptn\" (UniqueName: \"kubernetes.io/projected/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-api-access-blptn\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.601645 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.601544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.601645 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.601574 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702197 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702167 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blptn\" (UniqueName: \"kubernetes.io/projected/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-api-access-blptn\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702368 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702235 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702368 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702368 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc49631f-2768-44ba-815b-3603f99bd9ea-data-volume\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702368 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc49631f-2768-44ba-815b-3603f99bd9ea-crio-socket\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702368 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:37.702359 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:37.702561 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:37.702388 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:37.702561 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:37.702427 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:38.20241273 +0000 UTC m=+227.878474492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : configmap "kube-rbac-proxy" not found Jul 10 16:40:37.702561 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702428 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/dc49631f-2768-44ba-815b-3603f99bd9ea-crio-socket\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.702561 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:37.702451 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:38.202434749 +0000 UTC m=+227.878496514 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:37.702690 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.702621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc49631f-2768-44ba-815b-3603f99bd9ea-data-volume\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:37.711108 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:37.711081 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blptn\" (UniqueName: \"kubernetes.io/projected/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-api-access-blptn\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:38.206183 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:38.206151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:38.206183 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:38.206191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:38.206435 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:38.206267 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:38.206435 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:38.206302 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:38.206435 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:38.206325 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.20631115 +0000 UTC m=+228.882372911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : configmap "kube-rbac-proxy" not found Jul 10 16:40:38.206435 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:38.206361 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:39.20634589 +0000 UTC m=+228.882407667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:39.213764 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:39.213706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:39.213764 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:39.213771 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:39.214248 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:39.213840 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:39.214248 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:39.213925 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:41.213907969 +0000 UTC m=+230.889969731 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : configmap "kube-rbac-proxy" not found Jul 10 16:40:39.214248 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:39.213925 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:39.214248 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:39.213976 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:41.213961425 +0000 UTC m=+230.890023204 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:40.084316 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:40.084287 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ndkl_ff236cfe-018a-42db-a189-3b1d84b276cf/dns-node-resolver/0.log" Jul 10 16:40:41.085982 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:41.085948 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-692mv_a3758299-2838-43fd-ad6a-bc8b885579cd/node-ca/0.log" Jul 10 16:40:41.229146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:41.229110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:41.229146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:41.229146 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:41.229394 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:41.229237 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:41.229394 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:41.229283 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:45.229270819 +0000 UTC m=+234.905332585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:41.229394 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:41.229237 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:41.229394 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:41.229375 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:45.229354277 +0000 UTC m=+234.905416045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : configmap "kube-rbac-proxy" not found Jul 10 16:40:41.882980 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:41.882952 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2wq72_a74daaba-34e5-48e5-92b0-70066cce4536/serve-healthcheck-canary/0.log" Jul 10 16:40:45.261275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:45.261227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:45.261275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:45.261276 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:45.261873 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:45.261366 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 10 16:40:45.261873 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:45.261369 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 10 16:40:45.261873 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:45.261503 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:53.261480087 +0000 UTC m=+242.937541856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : configmap "kube-rbac-proxy" not found Jul 10 16:40:45.261873 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:40:45.261533 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls podName:dc49631f-2768-44ba-815b-3603f99bd9ea nodeName:}" failed. No retries permitted until 2026-07-10 16:40:53.261513459 +0000 UTC m=+242.937575221 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m4qbg" (UID: "dc49631f-2768-44ba-815b-3603f99bd9ea") : secret "insights-runtime-extractor-tls" not found Jul 10 16:40:53.320003 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.319962 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:53.320003 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.320001 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:53.320575 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.320542 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/dc49631f-2768-44ba-815b-3603f99bd9ea-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:53.322374 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.322349 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/dc49631f-2768-44ba-815b-3603f99bd9ea-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m4qbg\" (UID: \"dc49631f-2768-44ba-815b-3603f99bd9ea\") " pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:53.415627 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.415598 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2jlt9\"" Jul 10 16:40:53.423224 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.423202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m4qbg" Jul 10 16:40:53.544474 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.544442 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m4qbg"] Jul 10 16:40:53.547710 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:40:53.547682 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc49631f_2768_44ba_815b_3603f99bd9ea.slice/crio-b7303a261e24ab79f1dad32d4726de6febe76d42f30c53418d37fafa22d0fda5 WatchSource:0}: Error finding container b7303a261e24ab79f1dad32d4726de6febe76d42f30c53418d37fafa22d0fda5: Status 404 returned error can't find the container with id b7303a261e24ab79f1dad32d4726de6febe76d42f30c53418d37fafa22d0fda5 Jul 10 16:40:53.852730 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:53.852694 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4qbg" event={"ID":"dc49631f-2768-44ba-815b-3603f99bd9ea","Type":"ContainerStarted","Data":"b7303a261e24ab79f1dad32d4726de6febe76d42f30c53418d37fafa22d0fda5"} Jul 10 16:40:54.343000 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:54.342676 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" podUID="abae4655-7375-433c-a9ae-3264b9a2336b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 16:40:57.867630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:57.867597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4qbg" event={"ID":"dc49631f-2768-44ba-815b-3603f99bd9ea","Type":"ContainerStarted","Data":"0968e101de715651e715ab48a8ba8b1116b4ef1b7431488db6ac9bf0fc2bd413"} Jul 10 16:40:58.874050 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:58.874008 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4qbg" event={"ID":"dc49631f-2768-44ba-815b-3603f99bd9ea","Type":"ContainerStarted","Data":"4bb2cc45fd95f4b11ec85c784480f38a1349d84f9a1adaa7b4597f7556db58fa"} Jul 10 16:40:59.828577 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.828541 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-gxtzv"] Jul 10 16:40:59.836937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.836909 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:40:59.842896 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.842861 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-h45k9\"" Jul 10 16:40:59.843949 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.843770 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 10 16:40:59.844082 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.844027 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-gxtzv"] Jul 10 16:40:59.844551 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.844371 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 10 16:40:59.868766 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.868708 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnn5\" (UniqueName: \"kubernetes.io/projected/765508fe-517c-4c4a-a954-734c2573ede7-kube-api-access-jfnn5\") pod \"downloads-68c9cbb497-gxtzv\" (UID: \"765508fe-517c-4c4a-a954-734c2573ede7\") " pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:40:59.969889 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.969842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnn5\" (UniqueName: \"kubernetes.io/projected/765508fe-517c-4c4a-a954-734c2573ede7-kube-api-access-jfnn5\") pod \"downloads-68c9cbb497-gxtzv\" (UID: \"765508fe-517c-4c4a-a954-734c2573ede7\") " pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:40:59.981239 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:40:59.981207 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnn5\" (UniqueName: \"kubernetes.io/projected/765508fe-517c-4c4a-a954-734c2573ede7-kube-api-access-jfnn5\") pod \"downloads-68c9cbb497-gxtzv\" (UID: \"765508fe-517c-4c4a-a954-734c2573ede7\") " pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:41:00.148904 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:00.148824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:41:00.413325 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:00.413299 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-gxtzv"] Jul 10 16:41:00.416966 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:00.416936 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765508fe_517c_4c4a_a954_734c2573ede7.slice/crio-2d4415182c6357b3f2178e0c26bd65462b485db408f0acf5190bffc6e9493377 WatchSource:0}: Error finding container 2d4415182c6357b3f2178e0c26bd65462b485db408f0acf5190bffc6e9493377: Status 404 returned error can't find the container with id 2d4415182c6357b3f2178e0c26bd65462b485db408f0acf5190bffc6e9493377 Jul 10 16:41:00.881292 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:00.881250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-gxtzv" event={"ID":"765508fe-517c-4c4a-a954-734c2573ede7","Type":"ContainerStarted","Data":"2d4415182c6357b3f2178e0c26bd65462b485db408f0acf5190bffc6e9493377"} Jul 10 16:41:00.883158 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:00.883137 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m4qbg" event={"ID":"dc49631f-2768-44ba-815b-3603f99bd9ea","Type":"ContainerStarted","Data":"2298fd9e61b861050e1a00d53ec88bd2ef0212cd8711bcb923407a48ecef604a"} Jul 10 16:41:00.900947 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:00.900899 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m4qbg" podStartSLOduration=17.215775868 podStartE2EDuration="23.900882123s" totalImagesPullingTime="6.685106255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 16:40:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:00.900404535 +0000 UTC m=+250.576466320" watchObservedRunningTime="2026-07-10 16:41:00.900882123 +0000 UTC m=+250.576943907" Jul 10 16:41:02.789691 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:02.789649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:41:02.792359 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:02.792332 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cadf1967-b262-4bdd-a933-968447f88cf1-metrics-certs\") pod \"network-metrics-daemon-8mssf\" (UID: \"cadf1967-b262-4bdd-a933-968447f88cf1\") " pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:41:02.941944 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:02.941908 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bc5mm\"" Jul 10 16:41:02.949350 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:02.949325 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8mssf" Jul 10 16:41:03.096479 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:03.096405 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8mssf"] Jul 10 16:41:03.099518 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:03.099490 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadf1967_b262_4bdd_a933_968447f88cf1.slice/crio-2b0c7dc0316ffe5c530ab638073f3cfc1e759c508e451fbec68a674e1dca734c WatchSource:0}: Error finding container 2b0c7dc0316ffe5c530ab638073f3cfc1e759c508e451fbec68a674e1dca734c: Status 404 returned error can't find the container with id 2b0c7dc0316ffe5c530ab638073f3cfc1e759c508e451fbec68a674e1dca734c Jul 10 16:41:03.894613 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:03.894573 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mssf" event={"ID":"cadf1967-b262-4bdd-a933-968447f88cf1","Type":"ContainerStarted","Data":"2b0c7dc0316ffe5c530ab638073f3cfc1e759c508e451fbec68a674e1dca734c"} Jul 10 16:41:04.342590 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:04.342534 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" podUID="abae4655-7375-433c-a9ae-3264b9a2336b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 16:41:04.900439 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:04.900348 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mssf" event={"ID":"cadf1967-b262-4bdd-a933-968447f88cf1","Type":"ContainerStarted","Data":"93999c94a8c0c44eeef45340c3e725878d837252338f90892d3ccd7ed04276c5"} Jul 10 16:41:04.900439 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:04.900411 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8mssf" event={"ID":"cadf1967-b262-4bdd-a933-968447f88cf1","Type":"ContainerStarted","Data":"96437e57073c24b7a63c7f0b91730b5df968c3527f64ee154fd2d9f960f5610e"} Jul 10 16:41:04.917035 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:04.916983 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8mssf" podStartSLOduration=252.408956819 podStartE2EDuration="4m13.916961525s" totalImagesPullingTime="1.508004706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:36:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:04.915649098 +0000 UTC m=+254.591710896" watchObservedRunningTime="2026-07-10 16:41:04.916961525 +0000 UTC m=+254.593023309" Jul 10 16:41:08.549315 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.549278 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:08.552134 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.552114 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.555146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.554964 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 16:41:08.555146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.555022 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 16:41:08.555146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.554965 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b49fg\"" Jul 10 16:41:08.555146 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.555023 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 16:41:08.555421 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.554964 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 16:41:08.556132 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.556116 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 16:41:08.562336 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.562316 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:08.637719 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.637937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.637937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.637937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637877 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.637937 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.638152 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.637941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwqs7\" (UniqueName: \"kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739196 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739425 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739242 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739425 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739378 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739425 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739419 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739580 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.739580 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.739475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwqs7\" (UniqueName: \"kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.740063 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.740033 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.740198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.740078 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.740198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.740098 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.741678 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.741648 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.741812 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.741781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.750760 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.750716 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwqs7\" (UniqueName: \"kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7\") pod \"console-d98cd778c-zsg8m\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:08.863367 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:08.863285 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:09.025269 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:09.025237 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:09.027891 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:09.027864 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f3cfb7_8ed7_4e64_a5dd_d94539754087.slice/crio-48f80654be07187aa1841510f7556db6173ca4172f11a59049fe7e908cebbbd8 WatchSource:0}: Error finding container 48f80654be07187aa1841510f7556db6173ca4172f11a59049fe7e908cebbbd8: Status 404 returned error can't find the container with id 48f80654be07187aa1841510f7556db6173ca4172f11a59049fe7e908cebbbd8 Jul 10 16:41:09.918225 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:09.918186 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98cd778c-zsg8m" event={"ID":"65f3cfb7-8ed7-4e64-a5dd-d94539754087","Type":"ContainerStarted","Data":"48f80654be07187aa1841510f7556db6173ca4172f11a59049fe7e908cebbbd8"} Jul 10 16:41:12.929088 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:12.929053 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98cd778c-zsg8m" event={"ID":"65f3cfb7-8ed7-4e64-a5dd-d94539754087","Type":"ContainerStarted","Data":"5e43d92bf574b4dc8d80f296f20ad83752c8857526e3e215ae5a327b7bec3847"} Jul 10 16:41:12.948271 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:12.948167 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d98cd778c-zsg8m" podStartSLOduration=1.289541984 podStartE2EDuration="4.948147796s" totalImagesPullingTime="3.658605812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:12.946847971 +0000 UTC m=+262.622909758" watchObservedRunningTime="2026-07-10 16:41:12.948147796 +0000 UTC m=+262.624209580" Jul 10 16:41:14.343180 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:14.343124 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" podUID="abae4655-7375-433c-a9ae-3264b9a2336b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 10 16:41:14.343640 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:14.343239 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" Jul 10 16:41:14.343944 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:14.343892 2568 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8877fc09c36fe0b4fd4fd3766ae37d8ccba270fc7ed65a256f307676fd57fbf1"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 10 16:41:14.344019 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:14.343964 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" podUID="abae4655-7375-433c-a9ae-3264b9a2336b" containerName="service-proxy" containerID="cri-o://8877fc09c36fe0b4fd4fd3766ae37d8ccba270fc7ed65a256f307676fd57fbf1" gracePeriod=30 Jul 10 16:41:17.129050 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.129013 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fcs5s"] Jul 10 16:41:17.132592 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.132567 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.135403 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.135379 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 10 16:41:17.135573 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.135477 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 10 16:41:17.135665 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.135617 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 10 16:41:17.135923 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.135899 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5l4q2\"" Jul 10 16:41:17.136577 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.136560 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 10 16:41:17.136672 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.136607 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 10 16:41:17.136672 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.136667 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 10 16:41:17.213790 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.213756 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.213966 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.213809 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-root\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.213966 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.213882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-textfile\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.213966 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.213927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-sys\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.214111 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.213994 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4ld4\" (UniqueName: \"kubernetes.io/projected/a62d1948-0dfd-4404-ae42-daf90b9d5341-kube-api-access-c4ld4\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.214111 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.214022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.214111 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.214050 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-metrics-client-ca\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.214111 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.214088 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.214280 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.214113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-wtmp\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315237 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315194 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-root\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-textfile\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-sys\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4ld4\" (UniqueName: \"kubernetes.io/projected/a62d1948-0dfd-4404-ae42-daf90b9d5341-kube-api-access-c4ld4\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315340 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-root\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-metrics-client-ca\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315396 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-sys\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315440 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-wtmp\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315885 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315482 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315885 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:41:17.315582 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 10 16:41:17.315885 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:41:17.315661 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls podName:a62d1948-0dfd-4404-ae42-daf90b9d5341 nodeName:}" failed. No retries permitted until 2026-07-10 16:41:17.815641679 +0000 UTC m=+267.491703460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls") pod "node-exporter-fcs5s" (UID: "a62d1948-0dfd-4404-ae42-daf90b9d5341") : secret "node-exporter-tls" not found Jul 10 16:41:17.315885 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315733 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-textfile\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.315885 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.315866 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-wtmp\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.316151 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.316057 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-accelerators-collector-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.316374 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.316341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a62d1948-0dfd-4404-ae42-daf90b9d5341-metrics-client-ca\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.318197 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.318171 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.326324 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.326296 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4ld4\" (UniqueName: \"kubernetes.io/projected/a62d1948-0dfd-4404-ae42-daf90b9d5341-kube-api-access-c4ld4\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.820023 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.819986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:17.822617 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:17.822588 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a62d1948-0dfd-4404-ae42-daf90b9d5341-node-exporter-tls\") pod \"node-exporter-fcs5s\" (UID: \"a62d1948-0dfd-4404-ae42-daf90b9d5341\") " pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:18.043606 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:18.043570 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fcs5s" Jul 10 16:41:18.864452 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:18.864408 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:18.864951 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:18.864489 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:18.866179 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:18.866151 2568 patch_prober.go:28] interesting pod/console-d98cd778c-zsg8m container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" start-of-body= Jul 10 16:41:18.866310 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:18.866201 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-d98cd778c-zsg8m" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerName="console" probeResult="failure" output="Get \"https://10.134.0.12:8443/health\": dial tcp 10.134.0.12:8443: connect: connection refused" Jul 10 16:41:19.638478 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:19.638443 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62d1948_0dfd_4404_ae42_daf90b9d5341.slice/crio-1940e6e8d7c609b479a717320ff0919ed47620775455458a8e8d0a450a696808 WatchSource:0}: Error finding container 1940e6e8d7c609b479a717320ff0919ed47620775455458a8e8d0a450a696808: Status 404 returned error can't find the container with id 1940e6e8d7c609b479a717320ff0919ed47620775455458a8e8d0a450a696808 Jul 10 16:41:19.954465 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.954425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcs5s" event={"ID":"a62d1948-0dfd-4404-ae42-daf90b9d5341","Type":"ContainerStarted","Data":"1940e6e8d7c609b479a717320ff0919ed47620775455458a8e8d0a450a696808"} Jul 10 16:41:19.957149 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.957119 2568 generic.go:356] "Generic (PLEG): container finished" podID="abae4655-7375-433c-a9ae-3264b9a2336b" containerID="8877fc09c36fe0b4fd4fd3766ae37d8ccba270fc7ed65a256f307676fd57fbf1" exitCode=2 Jul 10 16:41:19.957269 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.957190 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerDied","Data":"8877fc09c36fe0b4fd4fd3766ae37d8ccba270fc7ed65a256f307676fd57fbf1"} Jul 10 16:41:19.957269 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.957242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7f784cd56-5cbn9" event={"ID":"abae4655-7375-433c-a9ae-3264b9a2336b","Type":"ContainerStarted","Data":"a6506ec457e652b4b647cb7be521a99cf0ec7f3b215a19334e19f7a4ed4fc8c5"} Jul 10 16:41:19.958713 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.958684 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-gxtzv" event={"ID":"765508fe-517c-4c4a-a954-734c2573ede7","Type":"ContainerStarted","Data":"49f515768f9ebcd1dc029218917dc14ae0a6d0e6301da53582627e8a1f891254"} Jul 10 16:41:19.959002 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.958966 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:41:19.967644 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.967602 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-gxtzv" Jul 10 16:41:19.993789 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:19.993712 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-gxtzv" podStartSLOduration=1.683954372 podStartE2EDuration="20.993695275s" totalImagesPullingTime="19.309740903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:40:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:19.993287157 +0000 UTC m=+269.669348943" watchObservedRunningTime="2026-07-10 16:41:19.993695275 +0000 UTC m=+269.669757060" Jul 10 16:41:20.964142 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:20.964100 2568 generic.go:356] "Generic (PLEG): container finished" podID="a62d1948-0dfd-4404-ae42-daf90b9d5341" containerID="56e4ec6af384134150e79875db454bff36ef84aefc0a6ed140d0a65d318619f4" exitCode=0 Jul 10 16:41:20.964608 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:20.964201 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcs5s" event={"ID":"a62d1948-0dfd-4404-ae42-daf90b9d5341","Type":"ContainerDied","Data":"56e4ec6af384134150e79875db454bff36ef84aefc0a6ed140d0a65d318619f4"} Jul 10 16:41:21.900522 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.900486 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:21.952852 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.952817 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:21.970306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.970089 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:21.970306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.970267 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcs5s" event={"ID":"a62d1948-0dfd-4404-ae42-daf90b9d5341","Type":"ContainerStarted","Data":"e75ada588e1482b4135f66af9b59b6909f8263a0bf41fc8790da3d5968b56fa4"} Jul 10 16:41:21.970926 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.970319 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fcs5s" event={"ID":"a62d1948-0dfd-4404-ae42-daf90b9d5341","Type":"ContainerStarted","Data":"aec66e2d09d5ce7ca200833d9bfa93369c0fc9819764bf71224fecb221bf9bc9"} Jul 10 16:41:21.973224 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.973198 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:21.986956 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:21.986925 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 16:41:22.006527 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.006469 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fcs5s" podStartSLOduration=3.9035457080000002 podStartE2EDuration="5.006447139s" totalImagesPullingTime="1.022701339s" totalInitContainerRuntime="80.200092ms" isStatefulPod=true podCreationTimestamp="2026-07-10 16:41:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:22.005637174 +0000 UTC m=+271.681698959" watchObservedRunningTime="2026-07-10 16:41:22.006447139 +0000 UTC m=+271.682508924" Jul 10 16:41:22.057970 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.057924 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058165 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.057983 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058165 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.058020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp8nf\" (UniqueName: \"kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058165 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.058081 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058165 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.058108 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058165 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.058151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.058484 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.058201 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.158915 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.158815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.158915 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.158899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159163 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.158949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159163 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.158982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159163 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.159037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp8nf\" (UniqueName: \"kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159163 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.159070 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159163 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.159096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.159653 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.159627 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.160155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.160128 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.160306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.160283 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.160444 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.160336 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.161945 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.161926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.162101 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.162079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.168315 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.168277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp8nf\" (UniqueName: \"kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf\") pod \"console-6cd6ff78c4-wgzsx\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.288460 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.288413 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:22.466570 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.466538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:22.469822 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:22.469790 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49554cc_a141_4395_824a_9589b81f2fa0.slice/crio-ad342976f875be45309a39e19018c2583b8336c5485cef5b667e1861fa04413f WatchSource:0}: Error finding container ad342976f875be45309a39e19018c2583b8336c5485cef5b667e1861fa04413f: Status 404 returned error can't find the container with id ad342976f875be45309a39e19018c2583b8336c5485cef5b667e1861fa04413f Jul 10 16:41:22.976505 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.976464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd6ff78c4-wgzsx" event={"ID":"d49554cc-a141-4395-824a-9589b81f2fa0","Type":"ContainerStarted","Data":"b88c0aa87a52366ff76ed4c044dce2d32cf3f6613e5a047cd369d23694cbc983"} Jul 10 16:41:22.976989 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.976527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd6ff78c4-wgzsx" event={"ID":"d49554cc-a141-4395-824a-9589b81f2fa0","Type":"ContainerStarted","Data":"ad342976f875be45309a39e19018c2583b8336c5485cef5b667e1861fa04413f"} Jul 10 16:41:22.994107 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:22.994047 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cd6ff78c4-wgzsx" podStartSLOduration=1.994028509 podStartE2EDuration="1.994028509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:22.993435646 +0000 UTC m=+272.669497445" watchObservedRunningTime="2026-07-10 16:41:22.994028509 +0000 UTC m=+272.670090292" Jul 10 16:41:27.876853 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:27.876816 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:32.289207 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:32.289167 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:33.617016 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:41:33.616966 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-8cdld" podUID="b397527c-1fa8-415d-ac0e-61dadab748f1" Jul 10 16:41:34.014944 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:34.014915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:36.580959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:36.580918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:36.583198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:36.583177 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b397527c-1fa8-415d-ac0e-61dadab748f1-metrics-tls\") pod \"dns-default-8cdld\" (UID: \"b397527c-1fa8-415d-ac0e-61dadab748f1\") " pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:36.718452 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:36.718417 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jqrx8\"" Jul 10 16:41:36.725841 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:36.725815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:37.060873 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:37.060850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8cdld"] Jul 10 16:41:37.063224 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:41:37.063195 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb397527c_1fa8_415d_ac0e_61dadab748f1.slice/crio-f64f585a891aa61301843712b818c802e23ac003fb22eb4ffe41ac6fbca591a2 WatchSource:0}: Error finding container f64f585a891aa61301843712b818c802e23ac003fb22eb4ffe41ac6fbca591a2: Status 404 returned error can't find the container with id f64f585a891aa61301843712b818c802e23ac003fb22eb4ffe41ac6fbca591a2 Jul 10 16:41:38.030403 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:38.030365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cdld" event={"ID":"b397527c-1fa8-415d-ac0e-61dadab748f1","Type":"ContainerStarted","Data":"f64f585a891aa61301843712b818c802e23ac003fb22eb4ffe41ac6fbca591a2"} Jul 10 16:41:41.041960 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:41.041930 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cdld" event={"ID":"b397527c-1fa8-415d-ac0e-61dadab748f1","Type":"ContainerStarted","Data":"28c224d2afdb0a61efc6a369a8f65925d89773cc61ea6418df3fa395f4fbcdb3"} Jul 10 16:41:41.042366 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:41.041974 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8cdld" event={"ID":"b397527c-1fa8-415d-ac0e-61dadab748f1","Type":"ContainerStarted","Data":"6b1f68ac0fb06e33f38594b302cb3a0d477a91a95044b3e6a96271077b7d0f2e"} Jul 10 16:41:41.042366 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:41.042065 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:41.063247 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:41.063197 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8cdld" podStartSLOduration=252.167111942 podStartE2EDuration="4m15.063179852s" totalImagesPullingTime="2.89606791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:41:41.062171384 +0000 UTC m=+290.738233170" watchObservedRunningTime="2026-07-10 16:41:41.063179852 +0000 UTC m=+290.739241635" Jul 10 16:41:46.925353 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:46.925284 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d98cd778c-zsg8m" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerName="console" containerID="cri-o://5e43d92bf574b4dc8d80f296f20ad83752c8857526e3e215ae5a327b7bec3847" gracePeriod=15 Jul 10 16:41:47.062715 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.062680 2568 generic.go:356] "Generic (PLEG): container finished" podID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerID="5e43d92bf574b4dc8d80f296f20ad83752c8857526e3e215ae5a327b7bec3847" exitCode=0 Jul 10 16:41:47.062897 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.062769 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98cd778c-zsg8m" event={"ID":"65f3cfb7-8ed7-4e64-a5dd-d94539754087","Type":"ContainerDied","Data":"5e43d92bf574b4dc8d80f296f20ad83752c8857526e3e215ae5a327b7bec3847"} Jul 10 16:41:47.163970 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.163944 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:47.256352 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256316 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwqs7\" (UniqueName: \"kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256352 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256352 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256612 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256404 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256612 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256426 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256612 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256448 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256612 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256474 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config\") pod \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\" (UID: \"65f3cfb7-8ed7-4e64-a5dd-d94539754087\") " Jul 10 16:41:47.256920 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256889 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca" (OuterVolumeSpecName: "service-ca") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:47.257032 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256910 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config" (OuterVolumeSpecName: "console-config") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:47.257032 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.256937 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:47.258702 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.258669 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7" (OuterVolumeSpecName: "kube-api-access-nwqs7") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "kube-api-access-nwqs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:41:47.259085 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.259033 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:47.259085 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.259044 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "65f3cfb7-8ed7-4e64-a5dd-d94539754087" (UID: "65f3cfb7-8ed7-4e64-a5dd-d94539754087"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:47.357632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357582 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-oauth-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:47.357632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357624 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:47.357632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357635 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-console-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:47.357632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357644 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwqs7\" (UniqueName: \"kubernetes.io/projected/65f3cfb7-8ed7-4e64-a5dd-d94539754087-kube-api-access-nwqs7\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:47.357632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357655 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-oauth-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:47.357961 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:47.357663 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65f3cfb7-8ed7-4e64-a5dd-d94539754087-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:48.066971 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:48.066932 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98cd778c-zsg8m" event={"ID":"65f3cfb7-8ed7-4e64-a5dd-d94539754087","Type":"ContainerDied","Data":"48f80654be07187aa1841510f7556db6173ca4172f11a59049fe7e908cebbbd8"} Jul 10 16:41:48.067377 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:48.067019 2568 scope.go:117] "RemoveContainer" containerID="5e43d92bf574b4dc8d80f296f20ad83752c8857526e3e215ae5a327b7bec3847" Jul 10 16:41:48.067377 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:48.066947 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98cd778c-zsg8m" Jul 10 16:41:48.086989 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:48.086962 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:48.089058 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:48.089037 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d98cd778c-zsg8m"] Jul 10 16:41:49.041991 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:49.041963 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" path="/var/lib/kubelet/pods/65f3cfb7-8ed7-4e64-a5dd-d94539754087/volumes" Jul 10 16:41:50.797661 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:50.797615 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:41:50.802077 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:50.801570 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:41:50.802881 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:50.802861 2568 kubelet.go:1648] "Image garbage collection succeeded" Jul 10 16:41:51.047302 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:51.047176 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8cdld" Jul 10 16:41:52.901601 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:52.901558 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cd6ff78c4-wgzsx" podUID="d49554cc-a141-4395-824a-9589b81f2fa0" containerName="console" containerID="cri-o://b88c0aa87a52366ff76ed4c044dce2d32cf3f6613e5a047cd369d23694cbc983" gracePeriod=15 Jul 10 16:41:53.086592 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.086544 2568 generic.go:356] "Generic (PLEG): container finished" podID="d49554cc-a141-4395-824a-9589b81f2fa0" containerID="b88c0aa87a52366ff76ed4c044dce2d32cf3f6613e5a047cd369d23694cbc983" exitCode=0 Jul 10 16:41:53.086592 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.086595 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd6ff78c4-wgzsx" event={"ID":"d49554cc-a141-4395-824a-9589b81f2fa0","Type":"ContainerDied","Data":"b88c0aa87a52366ff76ed4c044dce2d32cf3f6613e5a047cd369d23694cbc983"} Jul 10 16:41:53.177196 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.177123 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:53.193140 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193112 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp8nf\" (UniqueName: \"kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193140 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193143 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193161 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193201 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193229 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193260 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193306 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca\") pod \"d49554cc-a141-4395-824a-9589b81f2fa0\" (UID: \"d49554cc-a141-4395-824a-9589b81f2fa0\") " Jul 10 16:41:53.193727 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193701 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:53.193727 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193713 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:53.193882 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193697 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config" (OuterVolumeSpecName: "console-config") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:53.193934 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.193864 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca" (OuterVolumeSpecName: "service-ca") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:41:53.195587 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.195560 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf" (OuterVolumeSpecName: "kube-api-access-xp8nf") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "kube-api-access-xp8nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:41:53.195709 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.195665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:53.196182 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.196160 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d49554cc-a141-4395-824a-9589b81f2fa0" (UID: "d49554cc-a141-4395-824a-9589b81f2fa0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:41:53.293886 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293849 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-trusted-ca-bundle\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.293886 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293878 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-oauth-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.293886 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293887 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.293886 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293896 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp8nf\" (UniqueName: \"kubernetes.io/projected/d49554cc-a141-4395-824a-9589b81f2fa0-kube-api-access-xp8nf\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.294145 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293906 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.294145 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293928 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d49554cc-a141-4395-824a-9589b81f2fa0-console-oauth-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:53.294145 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:53.293937 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d49554cc-a141-4395-824a-9589b81f2fa0-console-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:41:54.091686 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:54.091659 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd6ff78c4-wgzsx" Jul 10 16:41:54.092149 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:54.091659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd6ff78c4-wgzsx" event={"ID":"d49554cc-a141-4395-824a-9589b81f2fa0","Type":"ContainerDied","Data":"ad342976f875be45309a39e19018c2583b8336c5485cef5b667e1861fa04413f"} Jul 10 16:41:54.092149 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:54.091807 2568 scope.go:117] "RemoveContainer" containerID="b88c0aa87a52366ff76ed4c044dce2d32cf3f6613e5a047cd369d23694cbc983" Jul 10 16:41:54.111518 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:54.111491 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:54.114075 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:54.114050 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd6ff78c4-wgzsx"] Jul 10 16:41:55.043270 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:41:55.043232 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49554cc-a141-4395-824a-9589b81f2fa0" path="/var/lib/kubelet/pods/d49554cc-a141-4395-824a-9589b81f2fa0/volumes" Jul 10 16:42:54.842615 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842582 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842809 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49554cc-a141-4395-824a-9589b81f2fa0" containerName="console" Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842821 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49554cc-a141-4395-824a-9589b81f2fa0" containerName="console" Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842831 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerName="console" Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842836 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerName="console" Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842883 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49554cc-a141-4395-824a-9589b81f2fa0" containerName="console" Jul 10 16:42:54.843068 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.842890 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f3cfb7-8ed7-4e64-a5dd-d94539754087" containerName="console" Jul 10 16:42:54.844858 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.844842 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.847593 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.847571 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 10 16:42:54.847715 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.847623 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 10 16:42:54.847782 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.847774 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 10 16:42:54.848884 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.848867 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 10 16:42:54.849151 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.849134 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 10 16:42:54.849201 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.849179 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-b49fg\"" Jul 10 16:42:54.853351 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.853334 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 10 16:42:54.863489 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.863460 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:42:54.908229 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908183 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j4ch\" (UniqueName: \"kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908333 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908357 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908375 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:54.908420 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:54.908413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009496 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009496 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009760 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009517 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009760 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009640 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009760 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009681 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009760 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009725 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.009957 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.009789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j4ch\" (UniqueName: \"kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.010332 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.010309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.010413 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.010309 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.010530 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.010508 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.010580 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.010560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.012082 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.012061 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.012156 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.012084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.018912 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.018889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j4ch\" (UniqueName: \"kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch\") pod \"console-5495d58c68-rw2mc\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.153798 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.153683 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:42:55.279468 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:55.279331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:42:55.281946 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:42:55.281917 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3943f969_5f9b_4f0a_a464_c5856333b7ed.slice/crio-e3ba5ada1b727a53ca1c709013c8c13e6471c713bf6c40079ec72780ddfc58e1 WatchSource:0}: Error finding container e3ba5ada1b727a53ca1c709013c8c13e6471c713bf6c40079ec72780ddfc58e1: Status 404 returned error can't find the container with id e3ba5ada1b727a53ca1c709013c8c13e6471c713bf6c40079ec72780ddfc58e1 Jul 10 16:42:56.294175 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:56.294138 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495d58c68-rw2mc" event={"ID":"3943f969-5f9b-4f0a-a464-c5856333b7ed","Type":"ContainerStarted","Data":"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e"} Jul 10 16:42:56.294687 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:56.294213 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495d58c68-rw2mc" event={"ID":"3943f969-5f9b-4f0a-a464-c5856333b7ed","Type":"ContainerStarted","Data":"e3ba5ada1b727a53ca1c709013c8c13e6471c713bf6c40079ec72780ddfc58e1"} Jul 10 16:42:56.313366 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:42:56.313309 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5495d58c68-rw2mc" podStartSLOduration=2.313294002 podStartE2EDuration="2.313294002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:42:56.312497706 +0000 UTC m=+365.988559500" watchObservedRunningTime="2026-07-10 16:42:56.313294002 +0000 UTC m=+365.989355786" Jul 10 16:43:05.154162 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:43:05.154107 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:43:05.154689 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:43:05.154195 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:43:05.158774 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:43:05.158732 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:43:05.326905 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:43:05.326880 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:44:39.415208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.415174 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z687s"] Jul 10 16:44:39.417075 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.417059 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.419988 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.419969 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 10 16:44:39.421060 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.421042 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-srjr5\"" Jul 10 16:44:39.421060 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.421054 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 10 16:44:39.427569 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.427546 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z687s"] Jul 10 16:44:39.561033 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.560999 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.561033 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.561041 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6hh\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-kube-api-access-kr6hh\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.662364 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.662325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.662364 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.662368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6hh\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-kube-api-access-kr6hh\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.673340 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.673277 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.673450 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.673369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6hh\" (UniqueName: \"kubernetes.io/projected/a4b99556-ac4c-4fbd-81b1-22498ecb0f96-kube-api-access-kr6hh\") pod \"cert-manager-cainjector-559cdd8c87-z687s\" (UID: \"a4b99556-ac4c-4fbd-81b1-22498ecb0f96\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.725035 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.725001 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" Jul 10 16:44:39.863330 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.863296 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-z687s"] Jul 10 16:44:39.866110 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:44:39.866080 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4b99556_ac4c_4fbd_81b1_22498ecb0f96.slice/crio-02f6226607d06954af41d05f7c5c8e385e1cba974bfb47eee417288197fb9f43 WatchSource:0}: Error finding container 02f6226607d06954af41d05f7c5c8e385e1cba974bfb47eee417288197fb9f43: Status 404 returned error can't find the container with id 02f6226607d06954af41d05f7c5c8e385e1cba974bfb47eee417288197fb9f43 Jul 10 16:44:39.867932 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:39.867918 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:44:40.633718 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:40.633679 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" event={"ID":"a4b99556-ac4c-4fbd-81b1-22498ecb0f96","Type":"ContainerStarted","Data":"02f6226607d06954af41d05f7c5c8e385e1cba974bfb47eee417288197fb9f43"} Jul 10 16:44:43.646134 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:43.646095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" event={"ID":"a4b99556-ac4c-4fbd-81b1-22498ecb0f96","Type":"ContainerStarted","Data":"b6222b5804af5a17ea90342aa1a76675dd2f1f6225754d7245743dc75b28892c"} Jul 10 16:44:43.662099 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:43.662049 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-z687s" podStartSLOduration=1.8309053629999998 podStartE2EDuration="4.66203278s" totalImagesPullingTime="2.831127417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:44:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:44:43.661566895 +0000 UTC m=+473.337628680" watchObservedRunningTime="2026-07-10 16:44:43.66203278 +0000 UTC m=+473.338094564" Jul 10 16:44:48.923707 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.923667 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm"] Jul 10 16:44:48.925777 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.925761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:48.928353 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.928330 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 10 16:44:48.929464 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.929450 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4brk6\"" Jul 10 16:44:48.929464 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.929459 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 10 16:44:48.933707 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:48.933679 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm"] Jul 10 16:44:49.037608 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.037570 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/388a8ca1-b816-4916-a106-a2c895480497-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.037608 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.037610 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t9v\" (UniqueName: \"kubernetes.io/projected/388a8ca1-b816-4916-a106-a2c895480497-kube-api-access-h7t9v\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.138901 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.138863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t9v\" (UniqueName: \"kubernetes.io/projected/388a8ca1-b816-4916-a106-a2c895480497-kube-api-access-h7t9v\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.139078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.138932 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/388a8ca1-b816-4916-a106-a2c895480497-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.139254 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.139238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/388a8ca1-b816-4916-a106-a2c895480497-tmp\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.147309 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.147280 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t9v\" (UniqueName: \"kubernetes.io/projected/388a8ca1-b816-4916-a106-a2c895480497-kube-api-access-h7t9v\") pod \"openshift-lws-operator-fd8ccff4c-sfrvm\" (UID: \"388a8ca1-b816-4916-a106-a2c895480497\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.234901 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.234878 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" Jul 10 16:44:49.361324 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.361301 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm"] Jul 10 16:44:49.363791 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:44:49.363761 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod388a8ca1_b816_4916_a106_a2c895480497.slice/crio-9faab56be1f1985097ea40ef2b28c724e07a72b2bc6e0bdefc17c8a114b1b0b9 WatchSource:0}: Error finding container 9faab56be1f1985097ea40ef2b28c724e07a72b2bc6e0bdefc17c8a114b1b0b9: Status 404 returned error can't find the container with id 9faab56be1f1985097ea40ef2b28c724e07a72b2bc6e0bdefc17c8a114b1b0b9 Jul 10 16:44:49.667885 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:49.667796 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" event={"ID":"388a8ca1-b816-4916-a106-a2c895480497","Type":"ContainerStarted","Data":"9faab56be1f1985097ea40ef2b28c724e07a72b2bc6e0bdefc17c8a114b1b0b9"} Jul 10 16:44:52.680933 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:52.680891 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" event={"ID":"388a8ca1-b816-4916-a106-a2c895480497","Type":"ContainerStarted","Data":"4ba1d24b2d1203be5c6d4e657696952886ada33cb56687acd7736476e98d2c0a"} Jul 10 16:44:52.698201 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:44:52.698148 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-sfrvm" podStartSLOduration=2.292527617 podStartE2EDuration="4.698131496s" totalImagesPullingTime="2.405603879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:44:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:44:52.697134672 +0000 UTC m=+482.373196459" watchObservedRunningTime="2026-07-10 16:44:52.698131496 +0000 UTC m=+482.374193280" Jul 10 16:45:09.292287 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.292253 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c"] Jul 10 16:45:09.294444 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.294420 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.297625 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.297602 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 10 16:45:09.297772 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.297602 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-v48n5\"" Jul 10 16:45:09.297772 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.297706 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 10 16:45:09.297971 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.297956 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 10 16:45:09.298049 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.297997 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 10 16:45:09.311118 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.311089 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c"] Jul 10 16:45:09.388312 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.388276 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.388495 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.388318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.388495 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.388341 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.388495 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.388421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8smd\" (UniqueName: \"kubernetes.io/projected/1adbb660-efa8-4bd2-aef0-ba129d2a011d-kube-api-access-d8smd\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.388495 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.388487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.489820 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.489785 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8smd\" (UniqueName: \"kubernetes.io/projected/1adbb660-efa8-4bd2-aef0-ba129d2a011d-kube-api-access-d8smd\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.490024 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.489845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.490024 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.489878 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.490024 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.489900 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.490024 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.489919 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.490349 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.490315 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-manifests\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.492174 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.492145 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1adbb660-efa8-4bd2-aef0-ba129d2a011d-tmp\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.492376 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.492359 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-webhook-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.492508 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.492487 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1adbb660-efa8-4bd2-aef0-ba129d2a011d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.498077 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.498054 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8smd\" (UniqueName: \"kubernetes.io/projected/1adbb660-efa8-4bd2-aef0-ba129d2a011d-kube-api-access-d8smd\") pod \"opendatahub-operator-controller-manager-585db49996-qmq6c\" (UID: \"1adbb660-efa8-4bd2-aef0-ba129d2a011d\") " pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.603818 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.603708 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:09.741206 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:09.741178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c"] Jul 10 16:45:09.744153 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:09.744122 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1adbb660_efa8_4bd2_aef0_ba129d2a011d.slice/crio-778a2496c5d98bdf244c3507e071abb0a774c758e3c401010be2ddc431f7cb32 WatchSource:0}: Error finding container 778a2496c5d98bdf244c3507e071abb0a774c758e3c401010be2ddc431f7cb32: Status 404 returned error can't find the container with id 778a2496c5d98bdf244c3507e071abb0a774c758e3c401010be2ddc431f7cb32 Jul 10 16:45:10.748155 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:10.748112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" event={"ID":"1adbb660-efa8-4bd2-aef0-ba129d2a011d","Type":"ContainerStarted","Data":"778a2496c5d98bdf244c3507e071abb0a774c758e3c401010be2ddc431f7cb32"} Jul 10 16:45:13.761082 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:13.761041 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" event={"ID":"1adbb660-efa8-4bd2-aef0-ba129d2a011d","Type":"ContainerStarted","Data":"cd35489f10dbacb57fb0aefd15e9e6494a2a4690b746840422ceaa298ff5e8dc"} Jul 10 16:45:14.766323 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:14.766288 2568 generic.go:356] "Generic (PLEG): container finished" podID="1adbb660-efa8-4bd2-aef0-ba129d2a011d" containerID="cd35489f10dbacb57fb0aefd15e9e6494a2a4690b746840422ceaa298ff5e8dc" exitCode=0 Jul 10 16:45:14.766705 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:14.766332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" event={"ID":"1adbb660-efa8-4bd2-aef0-ba129d2a011d","Type":"ContainerDied","Data":"cd35489f10dbacb57fb0aefd15e9e6494a2a4690b746840422ceaa298ff5e8dc"} Jul 10 16:45:15.772368 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:15.772337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" event={"ID":"1adbb660-efa8-4bd2-aef0-ba129d2a011d","Type":"ContainerStarted","Data":"be0b82192541a7586bc5df426c55271bf779688e65a40cfb51bcc45b1c2915a7"} Jul 10 16:45:15.772770 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:15.772464 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:15.795253 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:15.795196 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" podStartSLOduration=2.275963997 podStartE2EDuration="6.795178703s" totalImagesPullingTime="3.453520894s" totalInitContainerRuntime="1.065693812s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:15.793301054 +0000 UTC m=+505.469362839" watchObservedRunningTime="2026-07-10 16:45:15.795178703 +0000 UTC m=+505.471240486" Jul 10 16:45:26.778673 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:26.778636 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-585db49996-qmq6c" Jul 10 16:45:29.219186 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.219150 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gpz5"] Jul 10 16:45:29.229936 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.229907 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.232959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.232936 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gpz5"] Jul 10 16:45:29.232959 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.232941 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-5fwj2\"" Jul 10 16:45:29.233375 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.233358 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 10 16:45:29.347265 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.347225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.347265 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.347269 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4fp\" (UniqueName: \"kubernetes.io/projected/da915a25-efba-4b15-b7af-492200ad2c50-kube-api-access-rc4fp\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.448415 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.448377 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.448415 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.448419 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4fp\" (UniqueName: \"kubernetes.io/projected/da915a25-efba-4b15-b7af-492200ad2c50-kube-api-access-rc4fp\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.448615 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.448541 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 16:45:29.448658 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.448623 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert podName:da915a25-efba-4b15-b7af-492200ad2c50 nodeName:}" failed. No retries permitted until 2026-07-10 16:45:29.948600676 +0000 UTC m=+519.624662445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert") pod "odh-model-controller-7998c67dc4-8gpz5" (UID: "da915a25-efba-4b15-b7af-492200ad2c50") : secret "odh-model-controller-webhook-cert" not found Jul 10 16:45:29.465578 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.465553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4fp\" (UniqueName: \"kubernetes.io/projected/da915a25-efba-4b15-b7af-492200ad2c50-kube-api-access-rc4fp\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.716308 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.716271 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9r6j6"] Jul 10 16:45:29.719078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.719057 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:29.721983 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.721966 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 10 16:45:29.729715 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.729690 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9r6j6"] Jul 10 16:45:29.851151 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.851123 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8cc\" (UniqueName: \"kubernetes.io/projected/7832425b-88e9-4b4b-9447-66846bc99088-kube-api-access-9s8cc\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:29.851275 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.851189 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:29.952210 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.952171 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8cc\" (UniqueName: \"kubernetes.io/projected/7832425b-88e9-4b4b-9447-66846bc99088-kube-api-access-9s8cc\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:29.952210 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.952211 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:29.952410 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.952244 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:29.952410 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.952342 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 10 16:45:29.952410 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.952358 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 10 16:45:29.952410 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.952412 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs podName:7832425b-88e9-4b4b-9447-66846bc99088 nodeName:}" failed. No retries permitted until 2026-07-10 16:45:30.452396462 +0000 UTC m=+520.128458226 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs") pod "model-serving-api-54d89bd5fc-9r6j6" (UID: "7832425b-88e9-4b4b-9447-66846bc99088") : secret "model-serving-api-tls" not found Jul 10 16:45:29.952540 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:29.952425 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert podName:da915a25-efba-4b15-b7af-492200ad2c50 nodeName:}" failed. No retries permitted until 2026-07-10 16:45:30.952419473 +0000 UTC m=+520.628481235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert") pod "odh-model-controller-7998c67dc4-8gpz5" (UID: "da915a25-efba-4b15-b7af-492200ad2c50") : secret "odh-model-controller-webhook-cert" not found Jul 10 16:45:29.961599 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:29.961573 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8cc\" (UniqueName: \"kubernetes.io/projected/7832425b-88e9-4b4b-9447-66846bc99088-kube-api-access-9s8cc\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:30.457024 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.456984 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:30.459343 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.459323 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7832425b-88e9-4b4b-9447-66846bc99088-tls-certs\") pod \"model-serving-api-54d89bd5fc-9r6j6\" (UID: \"7832425b-88e9-4b4b-9447-66846bc99088\") " pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:30.629640 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.629598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:30.758800 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.758766 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-9r6j6"] Jul 10 16:45:30.763851 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:30.763824 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7832425b_88e9_4b4b_9447_66846bc99088.slice/crio-3a520c5d9c5958204e1c4cbb3cc35aac975bac519810199802cd9282e5af3173 WatchSource:0}: Error finding container 3a520c5d9c5958204e1c4cbb3cc35aac975bac519810199802cd9282e5af3173: Status 404 returned error can't find the container with id 3a520c5d9c5958204e1c4cbb3cc35aac975bac519810199802cd9282e5af3173 Jul 10 16:45:30.825191 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.825159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" event={"ID":"7832425b-88e9-4b4b-9447-66846bc99088","Type":"ContainerStarted","Data":"3a520c5d9c5958204e1c4cbb3cc35aac975bac519810199802cd9282e5af3173"} Jul 10 16:45:30.960138 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.960107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:30.962771 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:30.962686 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da915a25-efba-4b15-b7af-492200ad2c50-cert\") pod \"odh-model-controller-7998c67dc4-8gpz5\" (UID: \"da915a25-efba-4b15-b7af-492200ad2c50\") " pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:31.040981 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:31.040956 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:31.173431 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:31.173398 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8gpz5"] Jul 10 16:45:31.176088 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:31.176057 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda915a25_efba_4b15_b7af_492200ad2c50.slice/crio-3e62e463e2d9e44fe3dcac9f5ec94e9c933b27ac6d4c503e837e8e6782ea02f8 WatchSource:0}: Error finding container 3e62e463e2d9e44fe3dcac9f5ec94e9c933b27ac6d4c503e837e8e6782ea02f8: Status 404 returned error can't find the container with id 3e62e463e2d9e44fe3dcac9f5ec94e9c933b27ac6d4c503e837e8e6782ea02f8 Jul 10 16:45:31.829924 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:31.829880 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" event={"ID":"da915a25-efba-4b15-b7af-492200ad2c50","Type":"ContainerStarted","Data":"3e62e463e2d9e44fe3dcac9f5ec94e9c933b27ac6d4c503e837e8e6782ea02f8"} Jul 10 16:45:32.834658 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:32.834563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" event={"ID":"7832425b-88e9-4b4b-9447-66846bc99088","Type":"ContainerStarted","Data":"63f3a67a01df26e49d6363139f500beb3d5ebf708af902b45ea31754aee9d08c"} Jul 10 16:45:32.835109 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:32.834661 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:32.851607 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:32.851556 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" podStartSLOduration=2.065306332 podStartE2EDuration="3.851541213s" totalImagesPullingTime="1.786234881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:32.851172594 +0000 UTC m=+522.527234378" watchObservedRunningTime="2026-07-10 16:45:32.851541213 +0000 UTC m=+522.527602975" Jul 10 16:45:34.844469 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:34.844439 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" event={"ID":"da915a25-efba-4b15-b7af-492200ad2c50","Type":"ContainerStarted","Data":"d391c387a712198e149afd4eb4a2e2dcebab8ccc882aedf296242be5e80e33f6"} Jul 10 16:45:34.844857 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:34.844543 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:34.862113 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:34.862062 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" podStartSLOduration=2.787004222 podStartE2EDuration="5.862044877s" totalImagesPullingTime="3.075040655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:34.860651553 +0000 UTC m=+524.536713340" watchObservedRunningTime="2026-07-10 16:45:34.862044877 +0000 UTC m=+524.538106662" Jul 10 16:45:38.359193 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.359158 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb"] Jul 10 16:45:38.362239 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.362224 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.365281 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.365217 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 10 16:45:38.365281 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.365226 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-dqlb9\"" Jul 10 16:45:38.372842 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.372819 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb"] Jul 10 16:45:38.376201 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.376179 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5"] Jul 10 16:45:38.379187 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.379170 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:38.381671 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.381656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 10 16:45:38.381778 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.381734 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-l5pb5\"" Jul 10 16:45:38.390832 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.390805 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5"] Jul 10 16:45:38.420615 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.420579 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45spl\" (UniqueName: \"kubernetes.io/projected/9cd1b184-2cb5-4d92-a025-94923e4de14a-kube-api-access-45spl\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.420807 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.420623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.521115 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.521083 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45spl\" (UniqueName: \"kubernetes.io/projected/9cd1b184-2cb5-4d92-a025-94923e4de14a-kube-api-access-45spl\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.521290 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.521125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.521290 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.521179 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:38.521290 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.521196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/adc7a28b-4499-42dc-af21-f50804262a30-kube-api-access-jxkk9\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:38.521466 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:38.521293 2568 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 10 16:45:38.521466 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:38.521363 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert podName:9cd1b184-2cb5-4d92-a025-94923e4de14a nodeName:}" failed. No retries permitted until 2026-07-10 16:45:39.021342787 +0000 UTC m=+528.697404567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert") pod "kserve-controller-manager-d6f4b65b8-5h2lb" (UID: "9cd1b184-2cb5-4d92-a025-94923e4de14a") : secret "kserve-webhook-server-cert" not found Jul 10 16:45:38.535828 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.535802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45spl\" (UniqueName: \"kubernetes.io/projected/9cd1b184-2cb5-4d92-a025-94923e4de14a-kube-api-access-45spl\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:38.621627 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.621539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:38.621627 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.621577 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/adc7a28b-4499-42dc-af21-f50804262a30-kube-api-access-jxkk9\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:38.621842 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:38.621707 2568 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 10 16:45:38.621842 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:45:38.621805 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert podName:adc7a28b-4499-42dc-af21-f50804262a30 nodeName:}" failed. No retries permitted until 2026-07-10 16:45:39.121788902 +0000 UTC m=+528.797850668 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert") pod "llmisvc-controller-manager-777c6bd7d5-bq6n5" (UID: "adc7a28b-4499-42dc-af21-f50804262a30") : secret "llmisvc-webhook-server-cert" not found Jul 10 16:45:38.639830 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:38.639803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxkk9\" (UniqueName: \"kubernetes.io/projected/adc7a28b-4499-42dc-af21-f50804262a30-kube-api-access-jxkk9\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:39.025357 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.025322 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:39.027642 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.027622 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cd1b184-2cb5-4d92-a025-94923e4de14a-cert\") pod \"kserve-controller-manager-d6f4b65b8-5h2lb\" (UID: \"9cd1b184-2cb5-4d92-a025-94923e4de14a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:39.126579 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.126534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:39.129078 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.129046 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adc7a28b-4499-42dc-af21-f50804262a30-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bq6n5\" (UID: \"adc7a28b-4499-42dc-af21-f50804262a30\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:39.272318 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.272278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:39.289316 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.289292 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:39.449076 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.449036 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5"] Jul 10 16:45:39.453507 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:39.453366 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podadc7a28b_4499_42dc_af21_f50804262a30.slice/crio-1e997cb45e218e98ea3951027d22d9bc72984344d66176495c8c330691ca174c WatchSource:0}: Error finding container 1e997cb45e218e98ea3951027d22d9bc72984344d66176495c8c330691ca174c: Status 404 returned error can't find the container with id 1e997cb45e218e98ea3951027d22d9bc72984344d66176495c8c330691ca174c Jul 10 16:45:39.454248 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.454187 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb"] Jul 10 16:45:39.457720 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:39.457684 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1b184_2cb5_4d92_a025_94923e4de14a.slice/crio-e91ab76f0b0bbfb55f7c74a13fc9b5abace8e532e7a06e8e159462c03ca825a5 WatchSource:0}: Error finding container e91ab76f0b0bbfb55f7c74a13fc9b5abace8e532e7a06e8e159462c03ca825a5: Status 404 returned error can't find the container with id e91ab76f0b0bbfb55f7c74a13fc9b5abace8e532e7a06e8e159462c03ca825a5 Jul 10 16:45:39.616248 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.616172 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz"] Jul 10 16:45:39.620456 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.620434 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.649414 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.649389 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 10 16:45:39.649618 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.649604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 10 16:45:39.649678 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.649649 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 10 16:45:39.649678 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.649659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 10 16:45:39.733550 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.733513 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tls-certs\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.733550 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.733555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tmp\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.733766 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.733611 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grcd4\" (UniqueName: \"kubernetes.io/projected/8deaa765-28bb-4d0c-a71f-759bf8b6b639-kube-api-access-grcd4\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.738777 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.738730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz"] Jul 10 16:45:39.834514 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.834478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tls-certs\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.834514 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.834524 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tmp\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.834779 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.834548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grcd4\" (UniqueName: \"kubernetes.io/projected/8deaa765-28bb-4d0c-a71f-759bf8b6b639-kube-api-access-grcd4\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.836863 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.836838 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tmp\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.837070 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.837055 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8deaa765-28bb-4d0c-a71f-759bf8b6b639-tls-certs\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.843663 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.843637 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grcd4\" (UniqueName: \"kubernetes.io/projected/8deaa765-28bb-4d0c-a71f-759bf8b6b639-kube-api-access-grcd4\") pod \"kube-auth-proxy-5778d5b958-dwwbz\" (UID: \"8deaa765-28bb-4d0c-a71f-759bf8b6b639\") " pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:39.864797 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.864759 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" event={"ID":"adc7a28b-4499-42dc-af21-f50804262a30","Type":"ContainerStarted","Data":"1e997cb45e218e98ea3951027d22d9bc72984344d66176495c8c330691ca174c"} Jul 10 16:45:39.865906 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.865881 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" event={"ID":"9cd1b184-2cb5-4d92-a025-94923e4de14a","Type":"ContainerStarted","Data":"e91ab76f0b0bbfb55f7c74a13fc9b5abace8e532e7a06e8e159462c03ca825a5"} Jul 10 16:45:39.929941 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:39.929906 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" Jul 10 16:45:40.072758 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:40.072705 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz"] Jul 10 16:45:40.076322 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:40.076291 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8deaa765_28bb_4d0c_a71f_759bf8b6b639.slice/crio-bab0f347f87ecc6def95806b6482b0ab031472cd284711e9ef0d2f896bd350de WatchSource:0}: Error finding container bab0f347f87ecc6def95806b6482b0ab031472cd284711e9ef0d2f896bd350de: Status 404 returned error can't find the container with id bab0f347f87ecc6def95806b6482b0ab031472cd284711e9ef0d2f896bd350de Jul 10 16:45:40.873320 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:40.873276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" event={"ID":"8deaa765-28bb-4d0c-a71f-759bf8b6b639","Type":"ContainerStarted","Data":"bab0f347f87ecc6def95806b6482b0ab031472cd284711e9ef0d2f896bd350de"} Jul 10 16:45:41.605481 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.605398 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx"] Jul 10 16:45:41.608630 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.608615 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.612297 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.612276 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 10 16:45:41.612574 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.612552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-xgmh9\"" Jul 10 16:45:41.612861 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.612843 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 10 16:45:41.612926 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.612894 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 10 16:45:41.613123 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.613108 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 10 16:45:41.628171 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.628145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx"] Jul 10 16:45:41.751014 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.750970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751199 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751199 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4fc\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-kube-api-access-bm4fc\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751199 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751321 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751321 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751263 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.751321 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.751296 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852326 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4fc\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-kube-api-access-bm4fc\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852517 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852517 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852383 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852517 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852401 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852517 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852419 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852517 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852434 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.852832 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.852556 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.853084 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.853041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.855232 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.855201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.855341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.855206 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.855341 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.855204 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.855460 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.855341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.868216 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.867989 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.868353 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.868245 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4fc\" (UniqueName: \"kubernetes.io/projected/e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8-kube-api-access-bm4fc\") pod \"istiod-openshift-gateway-94bb8fbfd-sk8nx\" (UID: \"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:41.918277 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:41.918245 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:43.831906 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.831870 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx"] Jul 10 16:45:43.834646 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:45:43.834615 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f3223c_a28b_4e9c_9a7a_5b4efd9333c8.slice/crio-91f6f4f969e45b03520f34a69de4f8971e58782a2d6b856c1dc601e1657672e8 WatchSource:0}: Error finding container 91f6f4f969e45b03520f34a69de4f8971e58782a2d6b856c1dc601e1657672e8: Status 404 returned error can't find the container with id 91f6f4f969e45b03520f34a69de4f8971e58782a2d6b856c1dc601e1657672e8 Jul 10 16:45:43.843650 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.843565 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-9r6j6" Jul 10 16:45:43.892754 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.892705 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" event={"ID":"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8","Type":"ContainerStarted","Data":"91f6f4f969e45b03520f34a69de4f8971e58782a2d6b856c1dc601e1657672e8"} Jul 10 16:45:43.894409 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.894372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" event={"ID":"8deaa765-28bb-4d0c-a71f-759bf8b6b639","Type":"ContainerStarted","Data":"b1e869f3ff0fbb4aa8c7220422a03954bd393d2e31e3b591977a35465db3d9b5"} Jul 10 16:45:43.896066 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.896039 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" event={"ID":"adc7a28b-4499-42dc-af21-f50804262a30","Type":"ContainerStarted","Data":"889fb8e27b6af821ae637cbbeb9ac7a97bf10b5d424a4f3776cc5037a759ac9d"} Jul 10 16:45:43.896217 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.896157 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:45:43.897676 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.897647 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" event={"ID":"9cd1b184-2cb5-4d92-a025-94923e4de14a","Type":"ContainerStarted","Data":"6b01f9ea83812ea60c05858524c622f098288bddddcb7e0298c781813283b929"} Jul 10 16:45:43.897825 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.897799 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:45:43.912062 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.912007 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5778d5b958-dwwbz" podStartSLOduration=1.24199368 podStartE2EDuration="4.911990877s" totalImagesPullingTime="3.669997197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:43.910480386 +0000 UTC m=+533.586542171" watchObservedRunningTime="2026-07-10 16:45:43.911990877 +0000 UTC m=+533.588052660" Jul 10 16:45:43.928356 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.928308 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" podStartSLOduration=1.698253424 podStartE2EDuration="5.928292065s" totalImagesPullingTime="4.230038641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:43.928101854 +0000 UTC m=+533.604163638" watchObservedRunningTime="2026-07-10 16:45:43.928292065 +0000 UTC m=+533.604353848" Jul 10 16:45:43.945899 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:43.945808 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" podStartSLOduration=1.711643206 podStartE2EDuration="5.945790488s" totalImagesPullingTime="4.234147282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:43.944955453 +0000 UTC m=+533.621017239" watchObservedRunningTime="2026-07-10 16:45:43.945790488 +0000 UTC m=+533.621852273" Jul 10 16:45:45.850470 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:45.850435 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8gpz5" Jul 10 16:45:46.851577 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:46.851542 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 16:45:46.851902 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:46.851610 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 10 16:45:46.915996 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:46.915957 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" event={"ID":"e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8","Type":"ContainerStarted","Data":"10f37a6fb2465a6a05dbeae795de7c7d27bbda07967bd8218f7c9dc50e7b0b98"} Jul 10 16:45:46.916154 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:46.916139 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:45:46.937979 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:46.937929 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" podStartSLOduration=2.923540698 podStartE2EDuration="5.937911599s" totalImagesPullingTime="3.014370901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:45:46.935947401 +0000 UTC m=+536.612009184" watchObservedRunningTime="2026-07-10 16:45:46.937911599 +0000 UTC m=+536.613973383" Jul 10 16:45:47.921941 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:47.921908 2568 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-sk8nx container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 10 16:45:47.922431 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:47.921964 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" podUID="e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 10 16:45:50.922066 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:45:50.922033 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-sk8nx" Jul 10 16:46:14.904833 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:46:14.904800 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bq6n5" Jul 10 16:46:14.907687 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:46:14.907663 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5h2lb" Jul 10 16:46:45.660586 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:46:45.660548 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:46:50.831180 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:46:50.831152 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:46:50.834598 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:46:50.834578 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:47:10.680722 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.680653 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5495d58c68-rw2mc" podUID="3943f969-5f9b-4f0a-a464-c5856333b7ed" containerName="console" containerID="cri-o://ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e" gracePeriod=15 Jul 10 16:47:10.918310 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.918286 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:47:10.958247 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958164 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958247 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958197 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958471 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958258 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958471 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958318 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958471 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958343 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958471 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958368 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958471 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958398 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5j4ch\" (UniqueName: \"kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch\") pod \"3943f969-5f9b-4f0a-a464-c5856333b7ed\" (UID: \"3943f969-5f9b-4f0a-a464-c5856333b7ed\") " Jul 10 16:47:10.958715 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958625 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config" (OuterVolumeSpecName: "console-config") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:47:10.958806 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958733 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:47:10.958860 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958809 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca" (OuterVolumeSpecName: "service-ca") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:47:10.958860 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.958821 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 16:47:10.960469 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.960441 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:47:10.960568 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.960500 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 16:47:10.960568 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:10.960544 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch" (OuterVolumeSpecName: "kube-api-access-5j4ch") pod "3943f969-5f9b-4f0a-a464-c5856333b7ed" (UID: "3943f969-5f9b-4f0a-a464-c5856333b7ed"). InnerVolumeSpecName "kube-api-access-5j4ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:47:11.059001 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.058967 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059001 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.058999 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059001 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.059009 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-oauth-serving-cert\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.059018 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j4ch\" (UniqueName: \"kubernetes.io/projected/3943f969-5f9b-4f0a-a464-c5856333b7ed-kube-api-access-5j4ch\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.059027 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-trusted-ca-bundle\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.059036 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.059212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.059045 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3943f969-5f9b-4f0a-a464-c5856333b7ed-console-oauth-config\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:47:11.259537 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.259419 2568 generic.go:356] "Generic (PLEG): container finished" podID="3943f969-5f9b-4f0a-a464-c5856333b7ed" containerID="ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e" exitCode=0 Jul 10 16:47:11.259537 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.259465 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5495d58c68-rw2mc" Jul 10 16:47:11.259537 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.259499 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495d58c68-rw2mc" event={"ID":"3943f969-5f9b-4f0a-a464-c5856333b7ed","Type":"ContainerDied","Data":"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e"} Jul 10 16:47:11.259762 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.259543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5495d58c68-rw2mc" event={"ID":"3943f969-5f9b-4f0a-a464-c5856333b7ed","Type":"ContainerDied","Data":"e3ba5ada1b727a53ca1c709013c8c13e6471c713bf6c40079ec72780ddfc58e1"} Jul 10 16:47:11.259762 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.259562 2568 scope.go:117] "RemoveContainer" containerID="ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e" Jul 10 16:47:11.269417 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.269399 2568 scope.go:117] "RemoveContainer" containerID="ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e" Jul 10 16:47:11.269716 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:47:11.269698 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e\": container with ID starting with ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e not found: ID does not exist" containerID="ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e" Jul 10 16:47:11.269829 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.269724 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e"} err="failed to get container status \"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e\": rpc error: code = NotFound desc = could not find container \"ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e\": container with ID starting with ebb549933ef498f13dd68b83d87758b1514f4a2c387db2e732733120c7bc522e not found: ID does not exist" Jul 10 16:47:11.278023 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.277991 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:47:11.283839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:11.283815 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5495d58c68-rw2mc"] Jul 10 16:47:13.043321 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:13.043287 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3943f969-5f9b-4f0a-a464-c5856333b7ed" path="/var/lib/kubelet/pods/3943f969-5f9b-4f0a-a464-c5856333b7ed/volumes" Jul 10 16:47:21.312299 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.312260 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8m85g"] Jul 10 16:47:21.312831 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.312701 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3943f969-5f9b-4f0a-a464-c5856333b7ed" containerName="console" Jul 10 16:47:21.312831 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.312720 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3943f969-5f9b-4f0a-a464-c5856333b7ed" containerName="console" Jul 10 16:47:21.312831 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.312810 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3943f969-5f9b-4f0a-a464-c5856333b7ed" containerName="console" Jul 10 16:47:21.315352 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.315332 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:21.318235 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.318213 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 10 16:47:21.319573 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.319545 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 10 16:47:21.319674 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.319601 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-dfphp\"" Jul 10 16:47:21.324995 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.324966 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8m85g"] Jul 10 16:47:21.342632 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.342593 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwcxj\" (UniqueName: \"kubernetes.io/projected/0f810fb3-ebfc-4a93-a070-848831617a8c-kube-api-access-dwcxj\") pod \"authorino-operator-75cc8d5cb7-8m85g\" (UID: \"0f810fb3-ebfc-4a93-a070-848831617a8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:21.443109 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.443065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwcxj\" (UniqueName: \"kubernetes.io/projected/0f810fb3-ebfc-4a93-a070-848831617a8c-kube-api-access-dwcxj\") pod \"authorino-operator-75cc8d5cb7-8m85g\" (UID: \"0f810fb3-ebfc-4a93-a070-848831617a8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:21.451388 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.451357 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwcxj\" (UniqueName: \"kubernetes.io/projected/0f810fb3-ebfc-4a93-a070-848831617a8c-kube-api-access-dwcxj\") pod \"authorino-operator-75cc8d5cb7-8m85g\" (UID: \"0f810fb3-ebfc-4a93-a070-848831617a8c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:21.626906 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.626814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:21.755764 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:21.755583 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8m85g"] Jul 10 16:47:21.758558 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:47:21.758532 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f810fb3_ebfc_4a93_a070_848831617a8c.slice/crio-2ea67bb4c4429b0978fe3eefee6ab89ea4c267cfe9102ed67e22614ae33fb277 WatchSource:0}: Error finding container 2ea67bb4c4429b0978fe3eefee6ab89ea4c267cfe9102ed67e22614ae33fb277: Status 404 returned error can't find the container with id 2ea67bb4c4429b0978fe3eefee6ab89ea4c267cfe9102ed67e22614ae33fb277 Jul 10 16:47:22.306247 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:22.306211 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" event={"ID":"0f810fb3-ebfc-4a93-a070-848831617a8c","Type":"ContainerStarted","Data":"2ea67bb4c4429b0978fe3eefee6ab89ea4c267cfe9102ed67e22614ae33fb277"} Jul 10 16:47:24.317400 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:24.317365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" event={"ID":"0f810fb3-ebfc-4a93-a070-848831617a8c","Type":"ContainerStarted","Data":"5e72266e1423be99e339f8b4953e63dafa4e2969e5b30f39726a272c606d7061"} Jul 10 16:47:24.317878 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:24.317509 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:24.337524 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:24.337471 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" podStartSLOduration=0.921424692 podStartE2EDuration="3.337452989s" totalImagesPullingTime="2.416028297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:47:24.336477456 +0000 UTC m=+634.012539258" watchObservedRunningTime="2026-07-10 16:47:24.337452989 +0000 UTC m=+634.013514773" Jul 10 16:47:35.324401 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:35.324366 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8m85g" Jul 10 16:47:53.882578 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.882544 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:47:53.886720 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.886703 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:53.889454 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.889430 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8z57\"" Jul 10 16:47:53.899362 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.899340 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:47:53.996441 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.996402 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kln6h\" (UniqueName: \"kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:53.996616 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:53.996459 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.097840 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.097799 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.098010 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.097885 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kln6h\" (UniqueName: \"kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.098212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.098191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.107206 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.107175 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kln6h\" (UniqueName: \"kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h\") pod \"kuadrant-operator-controller-manager-86878c769f-2lbcl\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.196629 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.196594 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:47:54.328819 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.328785 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:47:54.332097 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:47:54.332066 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0efc62_1453_446c_9de1_d319ba8df45a.slice/crio-0494649e52381e10eae0a0beda200d09b79df239a882d8d2278480af80bbc00c WatchSource:0}: Error finding container 0494649e52381e10eae0a0beda200d09b79df239a882d8d2278480af80bbc00c: Status 404 returned error can't find the container with id 0494649e52381e10eae0a0beda200d09b79df239a882d8d2278480af80bbc00c Jul 10 16:47:54.441306 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:47:54.441269 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" event={"ID":"6d0efc62-1453-446c-9de1-d319ba8df45a","Type":"ContainerStarted","Data":"0494649e52381e10eae0a0beda200d09b79df239a882d8d2278480af80bbc00c"} Jul 10 16:48:00.471328 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:48:00.471295 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" event={"ID":"6d0efc62-1453-446c-9de1-d319ba8df45a","Type":"ContainerStarted","Data":"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042"} Jul 10 16:48:00.471729 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:48:00.471369 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:48:00.491446 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:48:00.491391 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" podStartSLOduration=2.176075344 podStartE2EDuration="7.491376579s" totalImagesPullingTime="5.315301235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:47:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:48:00.49010953 +0000 UTC m=+670.166171315" watchObservedRunningTime="2026-07-10 16:48:00.491376579 +0000 UTC m=+670.167438364" Jul 10 16:48:11.478209 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:48:11.478181 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:50:12.274627 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.274546 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8"] Jul 10 16:50:12.276732 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.276709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.279553 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.279524 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 10 16:50:12.280723 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.280706 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 10 16:50:12.280839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.280792 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kjfph\"" Jul 10 16:50:12.280839 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.280819 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 10 16:50:12.287114 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.287093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8"] Jul 10 16:50:12.402540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.402540 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402545 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.402820 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.402820 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.402820 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb04eb5-a83f-4526-8950-29246c34b620-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.402820 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.402647 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stntq\" (UniqueName: \"kubernetes.io/projected/cfb04eb5-a83f-4526-8950-29246c34b620-kube-api-access-stntq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503217 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503217 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503219 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503476 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503238 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503476 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503476 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb04eb5-a83f-4526-8950-29246c34b620-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503654 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stntq\" (UniqueName: \"kubernetes.io/projected/cfb04eb5-a83f-4526-8950-29246c34b620-kube-api-access-stntq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503654 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503639 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503804 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503693 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.503864 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.503809 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.505647 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.505622 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cfb04eb5-a83f-4526-8950-29246c34b620-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.505909 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.505891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cfb04eb5-a83f-4526-8950-29246c34b620-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.511415 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.511392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-stntq\" (UniqueName: \"kubernetes.io/projected/cfb04eb5-a83f-4526-8950-29246c34b620-kube-api-access-stntq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-b6sq8\" (UID: \"cfb04eb5-a83f-4526-8950-29246c34b620\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.587124 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.587030 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:12.713370 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.713331 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8"] Jul 10 16:50:12.718637 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:50:12.718616 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb04eb5_a83f_4526_8950_29246c34b620.slice/crio-87a0fde8b112be22805aa6eeb7f15053ca9d3d8ee40a62bd15b804293fdc37f6 WatchSource:0}: Error finding container 87a0fde8b112be22805aa6eeb7f15053ca9d3d8ee40a62bd15b804293fdc37f6: Status 404 returned error can't find the container with id 87a0fde8b112be22805aa6eeb7f15053ca9d3d8ee40a62bd15b804293fdc37f6 Jul 10 16:50:12.719977 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:12.719960 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 16:50:13.028110 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:13.028031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" event={"ID":"cfb04eb5-a83f-4526-8950-29246c34b620","Type":"ContainerStarted","Data":"87a0fde8b112be22805aa6eeb7f15053ca9d3d8ee40a62bd15b804293fdc37f6"} Jul 10 16:50:14.035264 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:14.035173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" event={"ID":"cfb04eb5-a83f-4526-8950-29246c34b620","Type":"ContainerStarted","Data":"224e4f46d83e8c323bd31c4c547c4cd9e4a17812203a1147795afb4bbbff9789"} Jul 10 16:50:14.055454 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:14.055402 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" podStartSLOduration=1.054933028 podStartE2EDuration="2.055388492s" totalImagesPullingTime="1.000455464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:50:14.054555092 +0000 UTC m=+803.730616875" watchObservedRunningTime="2026-07-10 16:50:14.055388492 +0000 UTC m=+803.731450276" Jul 10 16:50:22.587442 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:22.587390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:22.587967 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:22.587465 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:22.599926 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:22.599896 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:23.086138 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:23.086110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-b6sq8" Jul 10 16:50:39.596977 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.596945 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh"] Jul 10 16:50:39.600244 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.600226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.602791 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.602775 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 10 16:50:39.609605 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.609577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh"] Jul 10 16:50:39.716048 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716012 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11be1cc1-6075-480a-a975-6b0055f6e89f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.716212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716061 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.716212 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgbh\" (UniqueName: \"kubernetes.io/projected/11be1cc1-6075-480a-a975-6b0055f6e89f-kube-api-access-pvgbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.716283 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716227 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.716283 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.716283 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.716272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817422 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817599 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11be1cc1-6075-480a-a975-6b0055f6e89f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817599 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817599 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817525 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgbh\" (UniqueName: \"kubernetes.io/projected/11be1cc1-6075-480a-a975-6b0055f6e89f-kube-api-access-pvgbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817599 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817838 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.817962 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817935 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.818092 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.817998 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.818092 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.818032 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.819781 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.819762 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/11be1cc1-6075-480a-a975-6b0055f6e89f-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.819894 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.819876 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/11be1cc1-6075-480a-a975-6b0055f6e89f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.825208 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.825185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgbh\" (UniqueName: \"kubernetes.io/projected/11be1cc1-6075-480a-a975-6b0055f6e89f-kube-api-access-pvgbh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-cpffh\" (UID: \"11be1cc1-6075-480a-a975-6b0055f6e89f\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:39.911629 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:39.911549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:40.047198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:40.047168 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh"] Jul 10 16:50:40.049157 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:50:40.049128 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11be1cc1_6075_480a_a975_6b0055f6e89f.slice/crio-01cbb38ac5061f38540c4c2ddb2602146d1e39f678b2baaf520d60a93ee31d18 WatchSource:0}: Error finding container 01cbb38ac5061f38540c4c2ddb2602146d1e39f678b2baaf520d60a93ee31d18: Status 404 returned error can't find the container with id 01cbb38ac5061f38540c4c2ddb2602146d1e39f678b2baaf520d60a93ee31d18 Jul 10 16:50:40.145688 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:40.145654 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" event={"ID":"11be1cc1-6075-480a-a975-6b0055f6e89f","Type":"ContainerStarted","Data":"01cbb38ac5061f38540c4c2ddb2602146d1e39f678b2baaf520d60a93ee31d18"} Jul 10 16:50:41.151357 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:41.151318 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" event={"ID":"11be1cc1-6075-480a-a975-6b0055f6e89f","Type":"ContainerStarted","Data":"56b61ac2f885f4d05bad6522aa376158292c84816b5ca35e96bcc26cbde5d72b"} Jul 10 16:50:41.171223 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:41.171176 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" podStartSLOduration=1.997330356 podStartE2EDuration="2.171159337s" totalImagesPullingTime="173.828981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:50:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:50:41.169172741 +0000 UTC m=+830.845234538" watchObservedRunningTime="2026-07-10 16:50:41.171159337 +0000 UTC m=+830.847221121" Jul 10 16:50:49.911829 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:49.911799 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:49.912290 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:49.911868 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:49.924098 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:49.924074 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:50:50.200425 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:50:50.200402 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-cpffh" Jul 10 16:51:50.856714 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:51:50.856634 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:51:50.858895 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:51:50.858863 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:56:50.882160 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:56:50.882073 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:56:50.882785 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:56:50.882243 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 16:57:41.699461 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:41.699379 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:57:41.701947 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:41.699574 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" podUID="6d0efc62-1453-446c-9de1-d319ba8df45a" containerName="manager" containerID="cri-o://d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042" gracePeriod=10 Jul 10 16:57:44.547782 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.547752 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:57:44.671374 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.671289 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume\") pod \"6d0efc62-1453-446c-9de1-d319ba8df45a\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " Jul 10 16:57:44.671521 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.671377 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kln6h\" (UniqueName: \"kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h\") pod \"6d0efc62-1453-446c-9de1-d319ba8df45a\" (UID: \"6d0efc62-1453-446c-9de1-d319ba8df45a\") " Jul 10 16:57:44.671729 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.671705 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6d0efc62-1453-446c-9de1-d319ba8df45a" (UID: "6d0efc62-1453-446c-9de1-d319ba8df45a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 10 16:57:44.673488 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.673462 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h" (OuterVolumeSpecName: "kube-api-access-kln6h") pod "6d0efc62-1453-446c-9de1-d319ba8df45a" (UID: "6d0efc62-1453-446c-9de1-d319ba8df45a"). InnerVolumeSpecName "kube-api-access-kln6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 16:57:44.772907 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.772863 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d0efc62-1453-446c-9de1-d319ba8df45a-extensions-socket-volume\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:57:44.772907 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.772901 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kln6h\" (UniqueName: \"kubernetes.io/projected/6d0efc62-1453-446c-9de1-d319ba8df45a-kube-api-access-kln6h\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 16:57:44.942041 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.942011 2568 generic.go:356] "Generic (PLEG): container finished" podID="6d0efc62-1453-446c-9de1-d319ba8df45a" containerID="d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042" exitCode=0 Jul 10 16:57:44.942195 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.942053 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" Jul 10 16:57:44.942195 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.942068 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" event={"ID":"6d0efc62-1453-446c-9de1-d319ba8df45a","Type":"ContainerDied","Data":"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042"} Jul 10 16:57:44.942195 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.942112 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl" event={"ID":"6d0efc62-1453-446c-9de1-d319ba8df45a","Type":"ContainerDied","Data":"0494649e52381e10eae0a0beda200d09b79df239a882d8d2278480af80bbc00c"} Jul 10 16:57:44.942195 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.942132 2568 scope.go:117] "RemoveContainer" containerID="d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042" Jul 10 16:57:44.951307 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.951288 2568 scope.go:117] "RemoveContainer" containerID="d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042" Jul 10 16:57:44.951551 ip-10-0-131-107 kubenswrapper[2568]: E0710 16:57:44.951531 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042\": container with ID starting with d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042 not found: ID does not exist" containerID="d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042" Jul 10 16:57:44.951597 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.951560 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042"} err="failed to get container status \"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042\": rpc error: code = NotFound desc = could not find container \"d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042\": container with ID starting with d218e13929da831f473b5efe38647f0f1e70bb24b2f4c2522775734ded41c042 not found: ID does not exist" Jul 10 16:57:44.964347 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.964320 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:57:44.968106 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:44.968087 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-2lbcl"] Jul 10 16:57:45.043046 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:57:45.043015 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d0efc62-1453-446c-9de1-d319ba8df45a" path="/var/lib/kubelet/pods/6d0efc62-1453-446c-9de1-d319ba8df45a/volumes" Jul 10 16:58:42.802352 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.802320 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s"] Jul 10 16:58:42.802845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.802614 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d0efc62-1453-446c-9de1-d319ba8df45a" containerName="manager" Jul 10 16:58:42.802845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.802624 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d0efc62-1453-446c-9de1-d319ba8df45a" containerName="manager" Jul 10 16:58:42.802845 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.802675 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d0efc62-1453-446c-9de1-d319ba8df45a" containerName="manager" Jul 10 16:58:42.805571 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.805556 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:42.808472 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.808450 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8z57\"" Jul 10 16:58:42.818851 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.818826 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s"] Jul 10 16:58:42.911671 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.911629 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvhk7\" (UniqueName: \"kubernetes.io/projected/8b223535-8f86-485b-8052-28dcfd85f04c-kube-api-access-cvhk7\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:42.911873 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:42.911689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b223535-8f86-485b-8052-28dcfd85f04c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.013003 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.012958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvhk7\" (UniqueName: \"kubernetes.io/projected/8b223535-8f86-485b-8052-28dcfd85f04c-kube-api-access-cvhk7\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.013171 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.013022 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b223535-8f86-485b-8052-28dcfd85f04c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.013381 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.013364 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8b223535-8f86-485b-8052-28dcfd85f04c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.022333 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.022303 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvhk7\" (UniqueName: \"kubernetes.io/projected/8b223535-8f86-485b-8052-28dcfd85f04c-kube-api-access-cvhk7\") pod \"kuadrant-operator-controller-manager-86878c769f-b6h7s\" (UID: \"8b223535-8f86-485b-8052-28dcfd85f04c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.115678 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.115586 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:43.258780 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:43.258753 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s"] Jul 10 16:58:43.261291 ip-10-0-131-107 kubenswrapper[2568]: W0710 16:58:43.261260 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b223535_8f86_485b_8052_28dcfd85f04c.slice/crio-bf0b817d49ce44f35b5154f0f1a9426f19846b8dc62f1008bba2dbd2cb1280ce WatchSource:0}: Error finding container bf0b817d49ce44f35b5154f0f1a9426f19846b8dc62f1008bba2dbd2cb1280ce: Status 404 returned error can't find the container with id bf0b817d49ce44f35b5154f0f1a9426f19846b8dc62f1008bba2dbd2cb1280ce Jul 10 16:58:44.193426 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:44.193384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" event={"ID":"8b223535-8f86-485b-8052-28dcfd85f04c","Type":"ContainerStarted","Data":"e67da7df1aaf52736e9f027727d5697ef76252842a4a6ae621d1811f85075185"} Jul 10 16:58:44.193851 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:44.193438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" event={"ID":"8b223535-8f86-485b-8052-28dcfd85f04c","Type":"ContainerStarted","Data":"bf0b817d49ce44f35b5154f0f1a9426f19846b8dc62f1008bba2dbd2cb1280ce"} Jul 10 16:58:44.193851 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:44.193487 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 16:58:44.218875 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:44.218827 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" podStartSLOduration=2.218810697 podStartE2EDuration="2.218810697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 16:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 16:58:44.216806448 +0000 UTC m=+1313.892868232" watchObservedRunningTime="2026-07-10 16:58:44.218810697 +0000 UTC m=+1313.894872478" Jul 10 16:58:55.200198 ip-10-0-131-107 kubenswrapper[2568]: I0710 16:58:55.200172 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-b6h7s" Jul 10 17:00:00.142901 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.142869 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk"] Jul 10 17:00:00.146032 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.146015 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" Jul 10 17:00:00.148692 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.148670 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:00:00.151680 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.151651 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk"] Jul 10 17:00:00.156672 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.156654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" Jul 10 17:00:00.292061 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.291892 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk"] Jul 10 17:00:00.295223 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:00:00.295188 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6fc98fb_4ce1_437e_958f_dd8609341a28.slice/crio-b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e WatchSource:0}: Error finding container b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e: Status 404 returned error can't find the container with id b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e Jul 10 17:00:00.297135 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.297118 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:00:00.518509 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:00.518475 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" event={"ID":"d6fc98fb-4ce1-437e-958f-dd8609341a28","Type":"ContainerStarted","Data":"b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e"} Jul 10 17:00:04.537918 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:04.537877 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" event={"ID":"d6fc98fb-4ce1-437e-958f-dd8609341a28","Type":"ContainerStarted","Data":"afaed4a956116eda9a7c0a47a066ae03354d3ff0e13bbef6c17ab3ac5602159c"} Jul 10 17:00:04.554661 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:04.554610 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" podStartSLOduration=0.683516229 podStartE2EDuration="4.554593332s" totalImagesPullingTime="3.871077103s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:00:04.553395695 +0000 UTC m=+1394.229457480" watchObservedRunningTime="2026-07-10 17:00:04.554593332 +0000 UTC m=+1394.230655115" Jul 10 17:00:05.544506 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:05.544468 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6fc98fb-4ce1-437e-958f-dd8609341a28" containerID="afaed4a956116eda9a7c0a47a066ae03354d3ff0e13bbef6c17ab3ac5602159c" exitCode=0 Jul 10 17:00:05.544911 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:05.544556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" event={"ID":"d6fc98fb-4ce1-437e-958f-dd8609341a28","Type":"ContainerDied","Data":"afaed4a956116eda9a7c0a47a066ae03354d3ff0e13bbef6c17ab3ac5602159c"} Jul 10 17:00:06.680537 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:06.680509 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" Jul 10 17:00:07.554224 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:07.554194 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" Jul 10 17:00:07.554397 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:07.554224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728380-z8wpk" event={"ID":"d6fc98fb-4ce1-437e-958f-dd8609341a28","Type":"ContainerDied","Data":"b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e"} Jul 10 17:00:07.554397 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:00:07.554272 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b78e04b16f46351f2bb9b1238513a21a77e8883bbbe48b87950c747f8d5c8c0e" Jul 10 17:01:50.906029 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:01:50.905997 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:01:50.907244 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:01:50.907223 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:05:13.581511 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.581478 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:05:13.581933 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.581849 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6fc98fb-4ce1-437e-958f-dd8609341a28" containerName="cleanup" Jul 10 17:05:13.581933 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.581861 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6fc98fb-4ce1-437e-958f-dd8609341a28" containerName="cleanup" Jul 10 17:05:13.581933 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.581917 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6fc98fb-4ce1-437e-958f-dd8609341a28" containerName="cleanup" Jul 10 17:05:13.584693 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.584677 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.587468 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.587451 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:05:13.588734 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.588716 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:05:13.588864 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.588778 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7c0fd7dc-serving-cert\"" Jul 10 17:05:13.588864 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.588716 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:05:13.609223 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.609197 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:05:13.650979 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.650945 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56lfl\" (UniqueName: \"kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.651145 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.650997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.651145 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.651018 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.751712 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.751674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.751712 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.751711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.751932 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.751786 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56lfl\" (UniqueName: \"kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.752390 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.752365 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.754118 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.754092 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.768001 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.767975 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56lfl\" (UniqueName: \"kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl\") pod \"maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:13.894503 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:13.894417 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:14.058678 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:14.058646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:05:14.063232 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:05:14.063197 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8253dcb4_97ae_4e9a_bfda_877f3ef99ac8.slice/crio-a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe WatchSource:0}: Error finding container a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe: Status 404 returned error can't find the container with id a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe Jul 10 17:05:14.064454 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:14.064438 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:05:14.907160 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:14.907100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" event={"ID":"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8","Type":"ContainerStarted","Data":"a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe"} Jul 10 17:05:15.913953 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:15.913918 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" event={"ID":"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8","Type":"ContainerStarted","Data":"31045b04dc1386e647eac4f4739b86dd31494b756805e9583e4ed3b2d8151879"} Jul 10 17:05:15.914440 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:15.914024 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:15.951619 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:15.951563 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" podStartSLOduration=1.220751986 podStartE2EDuration="2.951545991s" totalImagesPullingTime="1.730794005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:05:15.95111192 +0000 UTC m=+1705.627173705" watchObservedRunningTime="2026-07-10 17:05:15.951545991 +0000 UTC m=+1705.627607816" Jul 10 17:05:18.479660 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.475187 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:18.479660 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.479584 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:18.482661 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.482637 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:05:18.488300 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.488270 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:18.598158 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.598120 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k\" (UID: \"878be6d2-1498-444c-b1a0-c3dc64865544\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:18.699310 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.699272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k\" (UID: \"878be6d2-1498-444c-b1a0-c3dc64865544\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:18.699932 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.699904 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k\" (UID: \"878be6d2-1498-444c-b1a0-c3dc64865544\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:18.791331 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.791243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:18.924665 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:18.924626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:18.927954 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:05:18.927929 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878be6d2_1498_444c_b1a0_c3dc64865544.slice/crio-0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2 WatchSource:0}: Error finding container 0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2: Status 404 returned error can't find the container with id 0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2 Jul 10 17:05:19.933797 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:19.933764 2568 generic.go:356] "Generic (PLEG): container finished" podID="878be6d2-1498-444c-b1a0-c3dc64865544" containerID="44d99d97b44a37a4192b6e81d1c539f0910f7e81ce962e93906e24b6c9bc3ee6" exitCode=7 Jul 10 17:05:19.934291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:19.933830 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" event={"ID":"878be6d2-1498-444c-b1a0-c3dc64865544","Type":"ContainerDied","Data":"44d99d97b44a37a4192b6e81d1c539f0910f7e81ce962e93906e24b6c9bc3ee6"} Jul 10 17:05:19.934291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:19.933887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" event={"ID":"878be6d2-1498-444c-b1a0-c3dc64865544","Type":"ContainerStarted","Data":"0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2"} Jul 10 17:05:19.934291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:19.934116 2568 scope.go:117] "RemoveContainer" containerID="44d99d97b44a37a4192b6e81d1c539f0910f7e81ce962e93906e24b6c9bc3ee6" Jul 10 17:05:20.940657 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:20.940618 2568 generic.go:356] "Generic (PLEG): container finished" podID="878be6d2-1498-444c-b1a0-c3dc64865544" containerID="4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac" exitCode=7 Jul 10 17:05:20.941132 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:20.940707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" event={"ID":"878be6d2-1498-444c-b1a0-c3dc64865544","Type":"ContainerDied","Data":"4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac"} Jul 10 17:05:20.941132 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:20.940798 2568 scope.go:117] "RemoveContainer" containerID="44d99d97b44a37a4192b6e81d1c539f0910f7e81ce962e93906e24b6c9bc3ee6" Jul 10 17:05:20.941132 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:20.940981 2568 scope.go:117] "RemoveContainer" containerID="4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac" Jul 10 17:05:20.941306 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:05:20.941169 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k_odh-ai-gateway-infra(878be6d2-1498-444c-b1a0-c3dc64865544)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" Jul 10 17:05:21.924281 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:21.924255 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:05:21.947424 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:21.947399 2568 scope.go:117] "RemoveContainer" containerID="4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac" Jul 10 17:05:21.947847 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:05:21.947577 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k_odh-ai-gateway-infra(878be6d2-1498-444c-b1a0-c3dc64865544)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" Jul 10 17:05:37.038598 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:37.038570 2568 scope.go:117] "RemoveContainer" containerID="4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac" Jul 10 17:05:38.024774 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:38.024677 2568 generic.go:356] "Generic (PLEG): container finished" podID="878be6d2-1498-444c-b1a0-c3dc64865544" containerID="1d7ac973cde65a633527ff6ddaa8cb3c20b86a1d927b2905011624fc221d83f9" exitCode=0 Jul 10 17:05:38.024774 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:38.024715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" event={"ID":"878be6d2-1498-444c-b1a0-c3dc64865544","Type":"ContainerDied","Data":"1d7ac973cde65a633527ff6ddaa8cb3c20b86a1d927b2905011624fc221d83f9"} Jul 10 17:05:38.024774 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:38.024776 2568 scope.go:117] "RemoveContainer" containerID="4b09ca7866f09eb5384a4d897c7a36e274c6b71df71f9d4db11c1a5ab14f05ac" Jul 10 17:05:39.056053 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:39.056020 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:39.170832 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:39.170806 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:39.278598 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:39.278569 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca\") pod \"878be6d2-1498-444c-b1a0-c3dc64865544\" (UID: \"878be6d2-1498-444c-b1a0-c3dc64865544\") " Jul 10 17:05:39.278996 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:39.278971 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "878be6d2-1498-444c-b1a0-c3dc64865544" (UID: "878be6d2-1498-444c-b1a0-c3dc64865544"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:05:39.379331 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:39.379243 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/878be6d2-1498-444c-b1a0-c3dc64865544-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:05:40.037014 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:40.036979 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" event={"ID":"878be6d2-1498-444c-b1a0-c3dc64865544","Type":"ContainerDied","Data":"0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2"} Jul 10 17:05:40.037175 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:40.037034 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0640c887f539766dba676c7609a0ad31b6be720e078a50f8830cc7c6ddfb35b2" Jul 10 17:05:40.037175 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:40.036997 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k" Jul 10 17:05:40.064871 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:05:40.064840 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod878be6d2_1498_444c_b1a0_c3dc64865544.slice\": RecentStats: unable to find data in memory cache]" Jul 10 17:05:40.073455 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:40.073425 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:40.077562 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:40.077540 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-rq42k"] Jul 10 17:05:41.047969 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:05:41.044329 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" path="/var/lib/kubelet/pods/878be6d2-1498-444c-b1a0-c3dc64865544/volumes" Jul 10 17:06:18.539785 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.539754 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6"] Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540097 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540110 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540124 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540129 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540200 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.540334 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.540209 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:18.543089 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.543074 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:18.547330 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.547308 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:06:18.553801 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.553771 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6"] Jul 10 17:06:18.617521 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.617481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6\" (UID: \"61d1f551-093b-482a-b151-d9e377cff94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:18.718623 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.718582 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6\" (UID: \"61d1f551-093b-482a-b151-d9e377cff94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:18.719274 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.719255 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6\" (UID: \"61d1f551-093b-482a-b151-d9e377cff94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:18.852779 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.852654 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:18.992096 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:18.992067 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6"] Jul 10 17:06:18.994469 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:06:18.994438 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d1f551_093b_482a_b151_d9e377cff94f.slice/crio-14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef WatchSource:0}: Error finding container 14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef: Status 404 returned error can't find the container with id 14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef Jul 10 17:06:19.208150 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:19.208106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" event={"ID":"61d1f551-093b-482a-b151-d9e377cff94f","Type":"ContainerStarted","Data":"b31a73dee89b0bcc21dfd13848cfb64079a54a0224ccadf8790f70250ad68c8e"} Jul 10 17:06:19.208322 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:19.208169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" event={"ID":"61d1f551-093b-482a-b151-d9e377cff94f","Type":"ContainerStarted","Data":"14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef"} Jul 10 17:06:19.228243 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:19.228186 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" podStartSLOduration=1.228163627 podStartE2EDuration="1.228163627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:19.227296367 +0000 UTC m=+1768.903358149" watchObservedRunningTime="2026-07-10 17:06:19.228163627 +0000 UTC m=+1768.904225413" Jul 10 17:06:20.213614 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:20.213571 2568 generic.go:356] "Generic (PLEG): container finished" podID="61d1f551-093b-482a-b151-d9e377cff94f" containerID="b31a73dee89b0bcc21dfd13848cfb64079a54a0224ccadf8790f70250ad68c8e" exitCode=0 Jul 10 17:06:20.214086 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:20.213659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" event={"ID":"61d1f551-093b-482a-b151-d9e377cff94f","Type":"ContainerDied","Data":"b31a73dee89b0bcc21dfd13848cfb64079a54a0224ccadf8790f70250ad68c8e"} Jul 10 17:06:21.351067 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:21.351041 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:21.443672 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:21.443636 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca\") pod \"61d1f551-093b-482a-b151-d9e377cff94f\" (UID: \"61d1f551-093b-482a-b151-d9e377cff94f\") " Jul 10 17:06:21.444047 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:21.444023 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61d1f551-093b-482a-b151-d9e377cff94f" (UID: "61d1f551-093b-482a-b151-d9e377cff94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:06:21.544610 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:21.544522 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61d1f551-093b-482a-b151-d9e377cff94f-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:06:22.222887 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:22.222862 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" Jul 10 17:06:22.223065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:22.222863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6" event={"ID":"61d1f551-093b-482a-b151-d9e377cff94f","Type":"ContainerDied","Data":"14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef"} Jul 10 17:06:22.223065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:22.222990 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14b640ef4acd99d746b2353c6933d36c1164e1dca5de82ccfd5e891654eec4ef" Jul 10 17:06:29.004732 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.004696 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:06:29.005831 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.005774 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" podUID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" containerName="maas-api" containerID="cri-o://31045b04dc1386e647eac4f4739b86dd31494b756805e9583e4ed3b2d8151879" gracePeriod=30 Jul 10 17:06:29.255219 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.255141 2568 generic.go:356] "Generic (PLEG): container finished" podID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" containerID="31045b04dc1386e647eac4f4739b86dd31494b756805e9583e4ed3b2d8151879" exitCode=0 Jul 10 17:06:29.255360 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.255214 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" event={"ID":"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8","Type":"ContainerDied","Data":"31045b04dc1386e647eac4f4739b86dd31494b756805e9583e4ed3b2d8151879"} Jul 10 17:06:29.255360 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.255283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" event={"ID":"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8","Type":"ContainerDied","Data":"a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe"} Jul 10 17:06:29.255360 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.255306 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a89e86ba306d3237604526439acebd1098546b88c497f5a0447b5fc645559abe" Jul 10 17:06:29.257190 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.257172 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:06:29.414983 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.414938 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56lfl\" (UniqueName: \"kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl\") pod \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " Jul 10 17:06:29.415152 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.415003 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca\") pod \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " Jul 10 17:06:29.415152 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.415057 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls\") pod \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\" (UID: \"8253dcb4-97ae-4e9a-bfda-877f3ef99ac8\") " Jul 10 17:06:29.415460 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.415430 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" (UID: "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:06:29.417210 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.417189 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" (UID: "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:06:29.417270 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.417216 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl" (OuterVolumeSpecName: "kube-api-access-56lfl") pod "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" (UID: "8253dcb4-97ae-4e9a-bfda-877f3ef99ac8"). InnerVolumeSpecName "kube-api-access-56lfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:06:29.516508 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.516416 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:06:29.516508 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.516448 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56lfl\" (UniqueName: \"kubernetes.io/projected/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-kube-api-access-56lfl\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:06:29.516508 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:29.516461 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:06:30.259772 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:30.259727 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn" Jul 10 17:06:30.280732 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:30.280700 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:06:30.285501 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:30.285456 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7c0fd7dc-5f7446d45c-r89rn"] Jul 10 17:06:31.043896 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:31.043866 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" path="/var/lib/kubelet/pods/8253dcb4-97ae-4e9a-bfda-877f3ef99ac8/volumes" Jul 10 17:06:45.403758 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.403699 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:06:45.404210 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404193 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61d1f551-093b-482a-b151-d9e377cff94f" containerName="revoke-keys" Jul 10 17:06:45.404287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404212 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="61d1f551-093b-482a-b151-d9e377cff94f" containerName="revoke-keys" Jul 10 17:06:45.404287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404236 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" containerName="maas-api" Jul 10 17:06:45.404287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404244 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" containerName="maas-api" Jul 10 17:06:45.404287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404260 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:45.404287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404269 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:45.404506 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404346 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="878be6d2-1498-444c-b1a0-c3dc64865544" containerName="revoke-keys" Jul 10 17:06:45.404506 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404357 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8253dcb4-97ae-4e9a-bfda-877f3ef99ac8" containerName="maas-api" Jul 10 17:06:45.404506 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.404367 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="61d1f551-093b-482a-b151-d9e377cff94f" containerName="revoke-keys" Jul 10 17:06:45.406785 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.406760 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.410391 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.410372 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:06:45.410783 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.410758 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fdbb8df2-serving-cert\"" Jul 10 17:06:45.410896 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.410763 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:06:45.411427 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.411408 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:06:45.420252 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.420227 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:06:45.447851 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.447816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.447851 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.447854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f96wh\" (UniqueName: \"kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.448063 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.447948 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.548470 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.548436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.548673 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.548487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.548673 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.548602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f96wh\" (UniqueName: \"kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.548828 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:06:45.548609 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-serving-cert: secret "maas-api-e2e-ait-fdbb8df2-serving-cert" not found Jul 10 17:06:45.548828 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:06:45.548761 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls podName:de109bdc-b330-45f2-8f0b-4ede4b9b4ecf nodeName:}" failed. No retries permitted until 2026-07-10 17:06:46.048723313 +0000 UTC m=+1795.724785075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls") pod "maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" (UID: "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf") : secret "maas-api-e2e-ait-fdbb8df2-serving-cert" not found Jul 10 17:06:45.549204 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.549185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:45.557541 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:45.557518 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f96wh\" (UniqueName: \"kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:46.052633 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:46.052597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:46.055065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:46.055046 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") pod \"maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:46.316732 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:46.316636 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:46.471563 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:46.471534 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:06:46.474313 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:06:46.474277 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde109bdc_b330_45f2_8f0b_4ede4b9b4ecf.slice/crio-d37ed082ba67d43c8a0d5dd39eebd767d0a4642fd87077a1d78faed2ca1aed07 WatchSource:0}: Error finding container d37ed082ba67d43c8a0d5dd39eebd767d0a4642fd87077a1d78faed2ca1aed07: Status 404 returned error can't find the container with id d37ed082ba67d43c8a0d5dd39eebd767d0a4642fd87077a1d78faed2ca1aed07 Jul 10 17:06:47.340777 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:47.340725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" event={"ID":"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf","Type":"ContainerStarted","Data":"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811"} Jul 10 17:06:47.340937 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:47.340799 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" event={"ID":"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf","Type":"ContainerStarted","Data":"d37ed082ba67d43c8a0d5dd39eebd767d0a4642fd87077a1d78faed2ca1aed07"} Jul 10 17:06:47.340937 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:47.340843 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:47.364650 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:47.364602 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" podStartSLOduration=1.649229635 podStartE2EDuration="2.364586333s" totalImagesPullingTime="715.356698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:47.363334849 +0000 UTC m=+1797.039396636" watchObservedRunningTime="2026-07-10 17:06:47.364586333 +0000 UTC m=+1797.040648116" Jul 10 17:06:50.343883 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.343847 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:06:50.347118 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.347102 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:06:50.350267 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.350246 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:06:50.356163 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.356133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:06:50.387956 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.387918 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht\" (UID: \"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:06:50.489106 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.489060 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht\" (UID: \"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:06:50.489713 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.489690 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht\" (UID: \"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:06:50.659259 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.659174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:06:50.791769 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.791720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:06:50.794810 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:06:50.794783 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0b57d_5aaa_4cae_8a67_0d0439b9e011.slice/crio-b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894 WatchSource:0}: Error finding container b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894: Status 404 returned error can't find the container with id b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894 Jul 10 17:06:50.929936 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.929913 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:06:50.932378 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:50.932351 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:06:51.365977 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:51.365887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerStarted","Data":"6eb4887c66a7ee51c64ba55d551036e3ac6f85ece81fba2c4e745bda3071361d"} Jul 10 17:06:51.365977 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:51.365970 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerStarted","Data":"b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894"} Jul 10 17:06:51.384175 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:51.384119 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" podStartSLOduration=1.384104549 podStartE2EDuration="1.384104549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:06:51.381815536 +0000 UTC m=+1801.057877321" watchObservedRunningTime="2026-07-10 17:06:51.384104549 +0000 UTC m=+1801.060166333" Jul 10 17:06:52.372235 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:52.372198 2568 generic.go:356] "Generic (PLEG): container finished" podID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerID="6eb4887c66a7ee51c64ba55d551036e3ac6f85ece81fba2c4e745bda3071361d" exitCode=7 Jul 10 17:06:52.372619 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:52.372271 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerDied","Data":"6eb4887c66a7ee51c64ba55d551036e3ac6f85ece81fba2c4e745bda3071361d"} Jul 10 17:06:52.372619 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:52.372498 2568 scope.go:117] "RemoveContainer" containerID="6eb4887c66a7ee51c64ba55d551036e3ac6f85ece81fba2c4e745bda3071361d" Jul 10 17:06:53.354255 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:53.354171 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:06:53.378611 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:53.378571 2568 generic.go:356] "Generic (PLEG): container finished" podID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerID="3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96" exitCode=7 Jul 10 17:06:53.378611 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:53.378612 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerDied","Data":"3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96"} Jul 10 17:06:53.379131 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:53.378662 2568 scope.go:117] "RemoveContainer" containerID="6eb4887c66a7ee51c64ba55d551036e3ac6f85ece81fba2c4e745bda3071361d" Jul 10 17:06:53.379131 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:53.378892 2568 scope.go:117] "RemoveContainer" containerID="3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96" Jul 10 17:06:53.379213 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:06:53.379143 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht_odh-ai-gateway-infra(b2f0b57d-5aaa-4cae-8a67-0d0439b9e011)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" Jul 10 17:06:54.385188 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:06:54.385162 2568 scope.go:117] "RemoveContainer" containerID="3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96" Jul 10 17:06:54.385629 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:06:54.385345 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht_odh-ai-gateway-infra(b2f0b57d-5aaa-4cae-8a67-0d0439b9e011)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" Jul 10 17:07:07.038185 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:07.038143 2568 scope.go:117] "RemoveContainer" containerID="3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96" Jul 10 17:07:07.446895 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:07.446858 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerStarted","Data":"8b35b865dfee52e70f90f0233745b69620bbc291a5ff738e4ff5400a3e9969e0"} Jul 10 17:07:08.453104 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:08.453073 2568 generic.go:356] "Generic (PLEG): container finished" podID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerID="8b35b865dfee52e70f90f0233745b69620bbc291a5ff738e4ff5400a3e9969e0" exitCode=0 Jul 10 17:07:08.453511 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:08.453147 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerDied","Data":"8b35b865dfee52e70f90f0233745b69620bbc291a5ff738e4ff5400a3e9969e0"} Jul 10 17:07:08.453511 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:08.453209 2568 scope.go:117] "RemoveContainer" containerID="3869f135c22d4a8e40a0d79c64932433c956a2aa5060d0f00042f6b2e9bedf96" Jul 10 17:07:08.482635 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:08.482605 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:07:09.593008 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:09.592979 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:07:09.646517 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:09.646485 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca\") pod \"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011\" (UID: \"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011\") " Jul 10 17:07:09.646887 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:09.646862 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" (UID: "b2f0b57d-5aaa-4cae-8a67-0d0439b9e011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:07:09.748042 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:09.747949 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:07:10.464693 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:10.464663 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" Jul 10 17:07:10.464693 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:10.464675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht" event={"ID":"b2f0b57d-5aaa-4cae-8a67-0d0439b9e011","Type":"ContainerDied","Data":"b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894"} Jul 10 17:07:10.464970 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:10.464767 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b24d2995a5c61059eb4b22572f44a6a1b4a5892f4c663071551b62f60d2fa894" Jul 10 17:07:10.497241 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:10.497199 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:07:10.500664 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:10.500633 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-r8jht"] Jul 10 17:07:11.042841 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:11.042811 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" path="/var/lib/kubelet/pods/b2f0b57d-5aaa-4cae-8a67-0d0439b9e011/volumes" Jul 10 17:07:50.399325 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399289 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk"] Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399608 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399618 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399636 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399642 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399651 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399656 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399714 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.399787 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.399722 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:07:50.402628 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.402612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:50.405957 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.405934 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:07:50.412365 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.412342 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk"] Jul 10 17:07:50.487234 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.487199 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk\" (UID: \"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:50.587571 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.587539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk\" (UID: \"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:50.588152 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.588134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk\" (UID: \"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:50.713541 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.713505 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:50.841243 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:50.841221 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk"] Jul 10 17:07:50.843778 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:07:50.843731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fc4c0b1_31d6_45b5_8cbe_5c2fbb27cf2a.slice/crio-9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532 WatchSource:0}: Error finding container 9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532: Status 404 returned error can't find the container with id 9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532 Jul 10 17:07:51.652397 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:51.652362 2568 generic.go:356] "Generic (PLEG): container finished" podID="4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a" containerID="c507ede20f1f3a6dd5aec13ca6a9087cce0b8b0c67c65eb79b6ab7b15bded6f5" exitCode=0 Jul 10 17:07:51.652799 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:51.652414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" event={"ID":"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a","Type":"ContainerDied","Data":"c507ede20f1f3a6dd5aec13ca6a9087cce0b8b0c67c65eb79b6ab7b15bded6f5"} Jul 10 17:07:51.652799 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:51.652470 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" event={"ID":"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a","Type":"ContainerStarted","Data":"9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532"} Jul 10 17:07:52.786716 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:52.786695 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:52.802767 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:52.802708 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca\") pod \"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a\" (UID: \"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a\") " Jul 10 17:07:52.803107 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:52.803083 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a" (UID: "4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:07:52.903637 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:52.903598 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:07:53.662535 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:53.662507 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" Jul 10 17:07:53.662752 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:53.662502 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk" event={"ID":"4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a","Type":"ContainerDied","Data":"9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532"} Jul 10 17:07:53.662752 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:07:53.662661 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a365e0f55329f2d30d4d98c4bcb329385afd3f02b26291fd4038ef591c07532" Jul 10 17:08:00.825125 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:00.825087 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:08:00.825550 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:00.825378 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" podUID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" containerName="maas-api" containerID="cri-o://72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811" gracePeriod=30 Jul 10 17:08:01.073344 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.073319 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:08:01.171670 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.171598 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f96wh\" (UniqueName: \"kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh\") pod \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " Jul 10 17:08:01.171670 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.171652 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") pod \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " Jul 10 17:08:01.171869 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.171830 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca\") pod \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\" (UID: \"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf\") " Jul 10 17:08:01.172181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.172151 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" (UID: "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:08:01.173643 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.173619 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh" (OuterVolumeSpecName: "kube-api-access-f96wh") pod "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" (UID: "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf"). InnerVolumeSpecName "kube-api-access-f96wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:08:01.173720 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.173698 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" (UID: "de109bdc-b330-45f2-8f0b-4ede4b9b4ecf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:08:01.273415 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.273380 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:08:01.273415 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.273407 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:08:01.273415 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.273416 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f96wh\" (UniqueName: \"kubernetes.io/projected/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf-kube-api-access-f96wh\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:08:01.699754 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.699699 2568 generic.go:356] "Generic (PLEG): container finished" podID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" containerID="72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811" exitCode=0 Jul 10 17:08:01.699964 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.699766 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" Jul 10 17:08:01.699964 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.699788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" event={"ID":"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf","Type":"ContainerDied","Data":"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811"} Jul 10 17:08:01.699964 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.699856 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp" event={"ID":"de109bdc-b330-45f2-8f0b-4ede4b9b4ecf","Type":"ContainerDied","Data":"d37ed082ba67d43c8a0d5dd39eebd767d0a4642fd87077a1d78faed2ca1aed07"} Jul 10 17:08:01.699964 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.699886 2568 scope.go:117] "RemoveContainer" containerID="72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811" Jul 10 17:08:01.708610 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.708594 2568 scope.go:117] "RemoveContainer" containerID="72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811" Jul 10 17:08:01.708872 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:08:01.708855 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811\": container with ID starting with 72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811 not found: ID does not exist" containerID="72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811" Jul 10 17:08:01.708928 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.708881 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811"} err="failed to get container status \"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811\": rpc error: code = NotFound desc = could not find container \"72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811\": container with ID starting with 72d633c12713937eb720311b358ade55460377d9ecabd67b61c428b1e1ec2811 not found: ID does not exist" Jul 10 17:08:01.722679 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.722658 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:08:01.727179 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:01.727160 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fdbb8df2-67784c6b87-mw6kp"] Jul 10 17:08:03.043942 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:03.043911 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" path="/var/lib/kubelet/pods/de109bdc-b330-45f2-8f0b-4ede4b9b4ecf/volumes" Jul 10 17:08:18.201277 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201245 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:08:18.201630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201551 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" containerName="maas-api" Jul 10 17:08:18.201630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201561 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" containerName="maas-api" Jul 10 17:08:18.201630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201569 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a" containerName="revoke-keys" Jul 10 17:08:18.201630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201575 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a" containerName="revoke-keys" Jul 10 17:08:18.201630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201632 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2f0b57d-5aaa-4cae-8a67-0d0439b9e011" containerName="revoke-keys" Jul 10 17:08:18.201824 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201641 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a" containerName="revoke-keys" Jul 10 17:08:18.201824 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.201648 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="de109bdc-b330-45f2-8f0b-4ede4b9b4ecf" containerName="maas-api" Jul 10 17:08:18.205687 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.205666 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.209286 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.209265 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ef0ffa22-serving-cert\"" Jul 10 17:08:18.209597 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.209580 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:08:18.209704 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.209627 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:08:18.210424 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.210406 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:08:18.220784 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.220758 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:08:18.324797 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.324730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.324797 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.324792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.325004 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.324845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt4mx\" (UniqueName: \"kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.425854 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.425821 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.425854 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.425859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.426069 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.425887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt4mx\" (UniqueName: \"kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.426069 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:08:18.425995 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-serving-cert: secret "maas-api-e2e-derive-ef0ffa22-serving-cert" not found Jul 10 17:08:18.426151 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:08:18.426076 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls podName:08815480-9a38-4546-acf1-0bfb09223cef nodeName:}" failed. No retries permitted until 2026-07-10 17:08:18.92605566 +0000 UTC m=+1888.602117435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls") pod "maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" (UID: "08815480-9a38-4546-acf1-0bfb09223cef") : secret "maas-api-e2e-derive-ef0ffa22-serving-cert" not found Jul 10 17:08:18.426453 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.426435 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.435894 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.435865 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt4mx\" (UniqueName: \"kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.929597 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.929561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:18.932255 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:18.932227 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") pod \"maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:19.115692 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:19.115655 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:19.255424 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:19.255397 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:08:19.257630 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:08:19.257603 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08815480_9a38_4546_acf1_0bfb09223cef.slice/crio-0f9feb9a42e8d4af57cf56ed4d7f81a085639f5a6b5dc7b45bf40f7bce95ad08 WatchSource:0}: Error finding container 0f9feb9a42e8d4af57cf56ed4d7f81a085639f5a6b5dc7b45bf40f7bce95ad08: Status 404 returned error can't find the container with id 0f9feb9a42e8d4af57cf56ed4d7f81a085639f5a6b5dc7b45bf40f7bce95ad08 Jul 10 17:08:19.782730 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:19.782698 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" event={"ID":"08815480-9a38-4546-acf1-0bfb09223cef","Type":"ContainerStarted","Data":"0f9feb9a42e8d4af57cf56ed4d7f81a085639f5a6b5dc7b45bf40f7bce95ad08"} Jul 10 17:08:20.788481 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:20.788448 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" event={"ID":"08815480-9a38-4546-acf1-0bfb09223cef","Type":"ContainerStarted","Data":"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88"} Jul 10 17:08:20.788886 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:20.788513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:20.807364 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:20.807315 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" podStartSLOduration=2.29037571 podStartE2EDuration="2.807297869s" totalImagesPullingTime="516.922159ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:08:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:08:20.806594577 +0000 UTC m=+1890.482656365" watchObservedRunningTime="2026-07-10 17:08:20.807297869 +0000 UTC m=+1890.483359653" Jul 10 17:08:22.344611 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.344576 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:22.347717 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.347700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:22.350364 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.350343 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:08:22.357183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.357159 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:22.455701 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.455665 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp\" (UID: \"f9ccc92c-91df-4ee0-be50-9e9371c3df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:22.556943 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.556903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp\" (UID: \"f9ccc92c-91df-4ee0-be50-9e9371c3df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:22.557534 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.557511 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp\" (UID: \"f9ccc92c-91df-4ee0-be50-9e9371c3df08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:22.658624 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.658537 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:22.789775 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.789728 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:22.793127 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:08:22.793100 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ccc92c_91df_4ee0_be50_9e9371c3df08.slice/crio-f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec WatchSource:0}: Error finding container f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec: Status 404 returned error can't find the container with id f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec Jul 10 17:08:22.799087 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:22.799062 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" event={"ID":"f9ccc92c-91df-4ee0-be50-9e9371c3df08","Type":"ContainerStarted","Data":"f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec"} Jul 10 17:08:23.804726 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:23.804640 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerID="f44ae3c87bc5e2dae265d6ed4c5e24b21ee66e9b8cd563a8090cf1cd10176bd7" exitCode=7 Jul 10 17:08:23.804726 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:23.804696 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" event={"ID":"f9ccc92c-91df-4ee0-be50-9e9371c3df08","Type":"ContainerDied","Data":"f44ae3c87bc5e2dae265d6ed4c5e24b21ee66e9b8cd563a8090cf1cd10176bd7"} Jul 10 17:08:23.805197 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:23.804959 2568 scope.go:117] "RemoveContainer" containerID="f44ae3c87bc5e2dae265d6ed4c5e24b21ee66e9b8cd563a8090cf1cd10176bd7" Jul 10 17:08:24.811589 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:24.811501 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerID="4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99" exitCode=7 Jul 10 17:08:24.811589 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:24.811563 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" event={"ID":"f9ccc92c-91df-4ee0-be50-9e9371c3df08","Type":"ContainerDied","Data":"4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99"} Jul 10 17:08:24.812065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:24.811635 2568 scope.go:117] "RemoveContainer" containerID="f44ae3c87bc5e2dae265d6ed4c5e24b21ee66e9b8cd563a8090cf1cd10176bd7" Jul 10 17:08:24.812065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:24.811845 2568 scope.go:117] "RemoveContainer" containerID="4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99" Jul 10 17:08:24.812065 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:08:24.812049 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp_odh-ai-gateway-infra(f9ccc92c-91df-4ee0-be50-9e9371c3df08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" Jul 10 17:08:25.818162 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:25.818137 2568 scope.go:117] "RemoveContainer" containerID="4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99" Jul 10 17:08:25.818575 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:08:25.818297 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp_odh-ai-gateway-infra(f9ccc92c-91df-4ee0-be50-9e9371c3df08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" Jul 10 17:08:26.798409 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:26.798378 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:08:38.038057 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:38.038024 2568 scope.go:117] "RemoveContainer" containerID="4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99" Jul 10 17:08:38.879616 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:38.879584 2568 generic.go:356] "Generic (PLEG): container finished" podID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerID="28665ab0a53da96cb6838a05cc05c2b8eb572678bd99deefdd9722ab2e03c127" exitCode=0 Jul 10 17:08:38.879854 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:38.879659 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" event={"ID":"f9ccc92c-91df-4ee0-be50-9e9371c3df08","Type":"ContainerDied","Data":"28665ab0a53da96cb6838a05cc05c2b8eb572678bd99deefdd9722ab2e03c127"} Jul 10 17:08:38.879854 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:38.879715 2568 scope.go:117] "RemoveContainer" containerID="4053ee590794f8206933f881fd68d851ba4f34aafce0cfe82a24965024c80a99" Jul 10 17:08:39.919945 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:39.919913 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:40.021396 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.021372 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:40.098288 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.098253 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca\") pod \"f9ccc92c-91df-4ee0-be50-9e9371c3df08\" (UID: \"f9ccc92c-91df-4ee0-be50-9e9371c3df08\") " Jul 10 17:08:40.098638 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.098617 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9ccc92c-91df-4ee0-be50-9e9371c3df08" (UID: "f9ccc92c-91df-4ee0-be50-9e9371c3df08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:08:40.199158 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.199126 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ccc92c-91df-4ee0-be50-9e9371c3df08-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:08:40.891538 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.891510 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" Jul 10 17:08:40.891710 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.891503 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp" event={"ID":"f9ccc92c-91df-4ee0-be50-9e9371c3df08","Type":"ContainerDied","Data":"f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec"} Jul 10 17:08:40.891710 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.891632 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9fe5ed2f09497631c191af35a84b0f7548ff6a6dd118430c5e6a47cc41d5aec" Jul 10 17:08:40.932303 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.932271 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:40.936755 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:40.936723 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-pxwgp"] Jul 10 17:08:41.043279 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:08:41.043249 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" path="/var/lib/kubelet/pods/f9ccc92c-91df-4ee0-be50-9e9371c3df08/volumes" Jul 10 17:09:12.383974 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.383944 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj"] Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384272 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384282 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384292 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384299 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384307 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384312 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384365 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.384456 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.384376 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:09:12.387235 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.387220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:12.389711 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.389679 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:09:12.394256 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.394203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj"] Jul 10 17:09:12.470280 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.470251 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj\" (UID: \"df36ef77-be30-4c1a-a515-b4c1fd35debf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:12.571636 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.571602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj\" (UID: \"df36ef77-be30-4c1a-a515-b4c1fd35debf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:12.572257 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.572233 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj\" (UID: \"df36ef77-be30-4c1a-a515-b4c1fd35debf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:12.697921 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.697873 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:12.824636 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:12.824612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj"] Jul 10 17:09:12.827115 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:09:12.827091 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf36ef77_be30_4c1a_a515_b4c1fd35debf.slice/crio-3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5 WatchSource:0}: Error finding container 3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5: Status 404 returned error can't find the container with id 3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5 Jul 10 17:09:13.037033 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:13.037002 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" event={"ID":"df36ef77-be30-4c1a-a515-b4c1fd35debf","Type":"ContainerStarted","Data":"3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5"} Jul 10 17:09:14.043143 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:14.043108 2568 generic.go:356] "Generic (PLEG): container finished" podID="df36ef77-be30-4c1a-a515-b4c1fd35debf" containerID="ffa5ddb264b96e9bc3563b97854751b8fc9f210c819ed395083930318e4eae03" exitCode=0 Jul 10 17:09:14.043512 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:14.043192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" event={"ID":"df36ef77-be30-4c1a-a515-b4c1fd35debf","Type":"ContainerDied","Data":"ffa5ddb264b96e9bc3563b97854751b8fc9f210c819ed395083930318e4eae03"} Jul 10 17:09:15.179960 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:15.179939 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:15.294930 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:15.294901 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca\") pod \"df36ef77-be30-4c1a-a515-b4c1fd35debf\" (UID: \"df36ef77-be30-4c1a-a515-b4c1fd35debf\") " Jul 10 17:09:15.295280 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:15.295257 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df36ef77-be30-4c1a-a515-b4c1fd35debf" (UID: "df36ef77-be30-4c1a-a515-b4c1fd35debf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:09:15.395593 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:15.395515 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df36ef77-be30-4c1a-a515-b4c1fd35debf-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:09:16.054213 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:16.054184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" Jul 10 17:09:16.054213 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:16.054198 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj" event={"ID":"df36ef77-be30-4c1a-a515-b4c1fd35debf","Type":"ContainerDied","Data":"3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5"} Jul 10 17:09:16.054419 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:16.054242 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db5ff750bf1311642adfff26197899609deafbe29b070acf83fdaa8769116f5" Jul 10 17:09:22.817013 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:22.816975 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:09:22.817586 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:22.817174 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" podUID="08815480-9a38-4546-acf1-0bfb09223cef" containerName="maas-api" containerID="cri-o://aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88" gracePeriod=30 Jul 10 17:09:23.069527 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.069473 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:09:23.088782 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.088696 2568 generic.go:356] "Generic (PLEG): container finished" podID="08815480-9a38-4546-acf1-0bfb09223cef" containerID="aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88" exitCode=0 Jul 10 17:09:23.088983 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.088779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" event={"ID":"08815480-9a38-4546-acf1-0bfb09223cef","Type":"ContainerDied","Data":"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88"} Jul 10 17:09:23.088983 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.088802 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" Jul 10 17:09:23.088983 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.088843 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8" event={"ID":"08815480-9a38-4546-acf1-0bfb09223cef","Type":"ContainerDied","Data":"0f9feb9a42e8d4af57cf56ed4d7f81a085639f5a6b5dc7b45bf40f7bce95ad08"} Jul 10 17:09:23.088983 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.088876 2568 scope.go:117] "RemoveContainer" containerID="aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88" Jul 10 17:09:23.097463 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.097446 2568 scope.go:117] "RemoveContainer" containerID="aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88" Jul 10 17:09:23.097770 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:09:23.097723 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88\": container with ID starting with aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88 not found: ID does not exist" containerID="aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88" Jul 10 17:09:23.097865 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.097779 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88"} err="failed to get container status \"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88\": rpc error: code = NotFound desc = could not find container \"aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88\": container with ID starting with aa11fdacedae7c96d0d2a1489d8f50a94c7f71cda2f555db028369327e667b88 not found: ID does not exist" Jul 10 17:09:23.163657 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.163624 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca\") pod \"08815480-9a38-4546-acf1-0bfb09223cef\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " Jul 10 17:09:23.163832 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.163667 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") pod \"08815480-9a38-4546-acf1-0bfb09223cef\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " Jul 10 17:09:23.163832 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.163789 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt4mx\" (UniqueName: \"kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx\") pod \"08815480-9a38-4546-acf1-0bfb09223cef\" (UID: \"08815480-9a38-4546-acf1-0bfb09223cef\") " Jul 10 17:09:23.164075 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.164040 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "08815480-9a38-4546-acf1-0bfb09223cef" (UID: "08815480-9a38-4546-acf1-0bfb09223cef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:09:23.164210 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.164143 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08815480-9a38-4546-acf1-0bfb09223cef-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:09:23.165808 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.165784 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx" (OuterVolumeSpecName: "kube-api-access-wt4mx") pod "08815480-9a38-4546-acf1-0bfb09223cef" (UID: "08815480-9a38-4546-acf1-0bfb09223cef"). InnerVolumeSpecName "kube-api-access-wt4mx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:09:23.165890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.165846 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "08815480-9a38-4546-acf1-0bfb09223cef" (UID: "08815480-9a38-4546-acf1-0bfb09223cef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:09:23.264559 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.264525 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08815480-9a38-4546-acf1-0bfb09223cef-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:09:23.264559 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.264553 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt4mx\" (UniqueName: \"kubernetes.io/projected/08815480-9a38-4546-acf1-0bfb09223cef-kube-api-access-wt4mx\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:09:23.410107 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.410073 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:09:23.416411 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:23.416379 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ef0ffa22-744c97b576-klnl8"] Jul 10 17:09:25.044417 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:09:25.044384 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08815480-9a38-4546-acf1-0bfb09223cef" path="/var/lib/kubelet/pods/08815480-9a38-4546-acf1-0bfb09223cef/volumes" Jul 10 17:11:16.788367 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788287 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:16.788916 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788810 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08815480-9a38-4546-acf1-0bfb09223cef" containerName="maas-api" Jul 10 17:11:16.788916 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788828 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="08815480-9a38-4546-acf1-0bfb09223cef" containerName="maas-api" Jul 10 17:11:16.788916 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788862 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df36ef77-be30-4c1a-a515-b4c1fd35debf" containerName="revoke-keys" Jul 10 17:11:16.788916 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788871 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="df36ef77-be30-4c1a-a515-b4c1fd35debf" containerName="revoke-keys" Jul 10 17:11:16.789053 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788950 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9ccc92c-91df-4ee0-be50-9e9371c3df08" containerName="revoke-keys" Jul 10 17:11:16.789053 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788965 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="df36ef77-be30-4c1a-a515-b4c1fd35debf" containerName="revoke-keys" Jul 10 17:11:16.789053 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.788978 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="08815480-9a38-4546-acf1-0bfb09223cef" containerName="maas-api" Jul 10 17:11:16.792204 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.792182 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.795601 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.795577 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e08efa8c-serving-cert\"" Jul 10 17:11:16.795601 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.795593 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:11:16.795790 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.795583 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:11:16.796510 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.796493 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:11:16.803503 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.803478 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:16.868727 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.868696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp27t\" (UniqueName: \"kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.868908 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.868770 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.868908 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.868829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.970063 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.970025 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.970236 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.970082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp27t\" (UniqueName: \"kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.970236 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.970126 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.970896 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.970872 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.972631 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.972602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:16.984330 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:16.984300 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp27t\" (UniqueName: \"kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t\") pod \"maas-api-e2e-mt-e08efa8c-b67bc484-pqfln\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:17.103172 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:17.103076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:17.242934 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:17.242908 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:17.245214 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:11:17.245184 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5fe7f73_ac03_4cae_8464_c0a94e20c772.slice/crio-c7f0c34ddc9502d287c367bd834ade2322df04165ee447092915fbf39556dc57 WatchSource:0}: Error finding container c7f0c34ddc9502d287c367bd834ade2322df04165ee447092915fbf39556dc57: Status 404 returned error can't find the container with id c7f0c34ddc9502d287c367bd834ade2322df04165ee447092915fbf39556dc57 Jul 10 17:11:17.246609 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:17.246582 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:11:17.628834 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:17.628795 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" event={"ID":"d5fe7f73-ac03-4cae-8464-c0a94e20c772","Type":"ContainerStarted","Data":"c7f0c34ddc9502d287c367bd834ade2322df04165ee447092915fbf39556dc57"} Jul 10 17:11:18.634427 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:18.634391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" event={"ID":"d5fe7f73-ac03-4cae-8464-c0a94e20c772","Type":"ContainerStarted","Data":"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace"} Jul 10 17:11:18.634910 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:18.634542 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:18.656984 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:18.656935 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" podStartSLOduration=2.1737268 podStartE2EDuration="2.656919747s" totalImagesPullingTime="483.192947ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:18.655416464 +0000 UTC m=+2068.331478250" watchObservedRunningTime="2026-07-10 17:11:18.656919747 +0000 UTC m=+2068.332981531" Jul 10 17:11:24.645311 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:24.645278 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:28.689354 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.689322 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm"] Jul 10 17:11:28.692489 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.692471 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:28.696185 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.696161 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:11:28.703348 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.703326 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm"] Jul 10 17:11:28.779391 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.779353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm\" (UID: \"dd2dc421-a448-474a-a2ec-1c82287bbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:28.886116 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.886057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm\" (UID: \"dd2dc421-a448-474a-a2ec-1c82287bbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:28.887135 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:28.887105 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm\" (UID: \"dd2dc421-a448-474a-a2ec-1c82287bbbd6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:29.003185 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:29.003083 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:29.140466 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:29.140431 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm"] Jul 10 17:11:29.142785 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:11:29.142731 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2dc421_a448_474a_a2ec_1c82287bbbd6.slice/crio-6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe WatchSource:0}: Error finding container 6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe: Status 404 returned error can't find the container with id 6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe Jul 10 17:11:29.686175 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:29.686140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" event={"ID":"dd2dc421-a448-474a-a2ec-1c82287bbbd6","Type":"ContainerStarted","Data":"0cf0bdca8c762ba37a1e381b7e86a6912750f98427ea586c2bfa31c3cd0f5170"} Jul 10 17:11:29.686370 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:29.686192 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" event={"ID":"dd2dc421-a448-474a-a2ec-1c82287bbbd6","Type":"ContainerStarted","Data":"6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe"} Jul 10 17:11:29.704559 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:29.704512 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" podStartSLOduration=1.7044973730000001 podStartE2EDuration="1.704497373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:11:29.702464932 +0000 UTC m=+2079.378526744" watchObservedRunningTime="2026-07-10 17:11:29.704497373 +0000 UTC m=+2079.380559156" Jul 10 17:11:30.692304 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:30.692269 2568 generic.go:356] "Generic (PLEG): container finished" podID="dd2dc421-a448-474a-a2ec-1c82287bbbd6" containerID="0cf0bdca8c762ba37a1e381b7e86a6912750f98427ea586c2bfa31c3cd0f5170" exitCode=0 Jul 10 17:11:30.692493 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:30.692313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" event={"ID":"dd2dc421-a448-474a-a2ec-1c82287bbbd6","Type":"ContainerDied","Data":"0cf0bdca8c762ba37a1e381b7e86a6912750f98427ea586c2bfa31c3cd0f5170"} Jul 10 17:11:31.826542 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:31.826515 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:31.911344 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:31.911313 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca\") pod \"dd2dc421-a448-474a-a2ec-1c82287bbbd6\" (UID: \"dd2dc421-a448-474a-a2ec-1c82287bbbd6\") " Jul 10 17:11:31.911696 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:31.911673 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dd2dc421-a448-474a-a2ec-1c82287bbbd6" (UID: "dd2dc421-a448-474a-a2ec-1c82287bbbd6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:11:32.012622 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:32.012539 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd2dc421-a448-474a-a2ec-1c82287bbbd6-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:11:32.702982 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:32.702945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" event={"ID":"dd2dc421-a448-474a-a2ec-1c82287bbbd6","Type":"ContainerDied","Data":"6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe"} Jul 10 17:11:32.702982 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:32.702973 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm" Jul 10 17:11:32.703239 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:32.703000 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6adcb12ec983268c735cfef5b3aeb1b51d7eaf32bf9d21ffabe48bfdc8c291fe" Jul 10 17:11:39.205932 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.205893 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:39.206461 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.206225 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" podUID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" containerName="maas-api" containerID="cri-o://4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace" gracePeriod=30 Jul 10 17:11:39.455856 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.455833 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:39.574277 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.574182 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mp27t\" (UniqueName: \"kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t\") pod \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " Jul 10 17:11:39.574277 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.574228 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls\") pod \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " Jul 10 17:11:39.574492 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.574297 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca\") pod \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\" (UID: \"d5fe7f73-ac03-4cae-8464-c0a94e20c772\") " Jul 10 17:11:39.574695 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.574667 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d5fe7f73-ac03-4cae-8464-c0a94e20c772" (UID: "d5fe7f73-ac03-4cae-8464-c0a94e20c772"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:11:39.576330 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.576302 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d5fe7f73-ac03-4cae-8464-c0a94e20c772" (UID: "d5fe7f73-ac03-4cae-8464-c0a94e20c772"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:11:39.576408 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.576354 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t" (OuterVolumeSpecName: "kube-api-access-mp27t") pod "d5fe7f73-ac03-4cae-8464-c0a94e20c772" (UID: "d5fe7f73-ac03-4cae-8464-c0a94e20c772"). InnerVolumeSpecName "kube-api-access-mp27t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:11:39.675370 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.675335 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp27t\" (UniqueName: \"kubernetes.io/projected/d5fe7f73-ac03-4cae-8464-c0a94e20c772-kube-api-access-mp27t\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:11:39.675370 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.675365 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5fe7f73-ac03-4cae-8464-c0a94e20c772-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:11:39.675370 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.675375 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5fe7f73-ac03-4cae-8464-c0a94e20c772-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:11:39.737588 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.737551 2568 generic.go:356] "Generic (PLEG): container finished" podID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" containerID="4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace" exitCode=0 Jul 10 17:11:39.737762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.737592 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" Jul 10 17:11:39.737762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.737624 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" event={"ID":"d5fe7f73-ac03-4cae-8464-c0a94e20c772","Type":"ContainerDied","Data":"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace"} Jul 10 17:11:39.737762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.737678 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln" event={"ID":"d5fe7f73-ac03-4cae-8464-c0a94e20c772","Type":"ContainerDied","Data":"c7f0c34ddc9502d287c367bd834ade2322df04165ee447092915fbf39556dc57"} Jul 10 17:11:39.737762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.737702 2568 scope.go:117] "RemoveContainer" containerID="4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace" Jul 10 17:11:39.747293 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.747271 2568 scope.go:117] "RemoveContainer" containerID="4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace" Jul 10 17:11:39.747585 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:11:39.747562 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace\": container with ID starting with 4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace not found: ID does not exist" containerID="4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace" Jul 10 17:11:39.747629 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.747595 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace"} err="failed to get container status \"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace\": rpc error: code = NotFound desc = could not find container \"4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace\": container with ID starting with 4117f541d6149740f4f168b36274528054589dfd72e7ef407e91dccc75e39ace not found: ID does not exist" Jul 10 17:11:39.761147 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.761120 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:39.767133 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:39.767104 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e08efa8c-b67bc484-pqfln"] Jul 10 17:11:41.043604 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:41.043570 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" path="/var/lib/kubelet/pods/d5fe7f73-ac03-4cae-8464-c0a94e20c772/volumes" Jul 10 17:11:50.958080 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:50.958053 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:11:50.960733 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:50.960709 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:11:51.016749 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:51.016719 2568 scope.go:117] "RemoveContainer" containerID="31045b04dc1386e647eac4f4739b86dd31494b756805e9583e4ed3b2d8151879" Jul 10 17:11:51.024344 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:11:51.024322 2568 scope.go:117] "RemoveContainer" containerID="1d7ac973cde65a633527ff6ddaa8cb3c20b86a1d927b2905011624fc221d83f9" Jul 10 17:12:17.095966 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.095935 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096230 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" containerName="maas-api" Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096240 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" containerName="maas-api" Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096261 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd2dc421-a448-474a-a2ec-1c82287bbbd6" containerName="revoke-keys" Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096266 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd2dc421-a448-474a-a2ec-1c82287bbbd6" containerName="revoke-keys" Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096314 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5fe7f73-ac03-4cae-8464-c0a94e20c772" containerName="maas-api" Jul 10 17:12:17.096333 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.096322 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd2dc421-a448-474a-a2ec-1c82287bbbd6" containerName="revoke-keys" Jul 10 17:12:17.099489 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.099472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.108093 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.108062 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:12:17.108201 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.108137 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-af50a058-serving-cert\"" Jul 10 17:12:17.109068 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.109053 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:12:17.110578 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.110563 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:12:17.121318 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.121290 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:12:17.187324 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.187282 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hhgb\" (UniqueName: \"kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.187719 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.187696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.187830 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.187730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.288652 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.288616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.288652 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.288655 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.288875 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.288688 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hhgb\" (UniqueName: \"kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.288875 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:12:17.288816 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-serving-cert: secret "maas-api-e2e-mt-af50a058-serving-cert" not found Jul 10 17:12:17.288942 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:12:17.288906 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls podName:64369252-889a-4116-be41-95bc8d95a4e7 nodeName:}" failed. No retries permitted until 2026-07-10 17:12:17.788883763 +0000 UTC m=+2127.464945535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls") pod "maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" (UID: "64369252-889a-4116-be41-95bc8d95a4e7") : secret "maas-api-e2e-mt-af50a058-serving-cert" not found Jul 10 17:12:17.289397 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.289379 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.303181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.303153 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hhgb\" (UniqueName: \"kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.793285 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.793248 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:17.795642 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:17.795621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") pod \"maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:18.008992 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.008952 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:18.145759 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.145712 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:12:18.148847 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:12:18.148818 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64369252_889a_4116_be41_95bc8d95a4e7.slice/crio-529fadde653674bf5132ae29c7b8ba60aef375b5f5e80d7a3d4b9cbbe4952ee1 WatchSource:0}: Error finding container 529fadde653674bf5132ae29c7b8ba60aef375b5f5e80d7a3d4b9cbbe4952ee1: Status 404 returned error can't find the container with id 529fadde653674bf5132ae29c7b8ba60aef375b5f5e80d7a3d4b9cbbe4952ee1 Jul 10 17:12:18.916690 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.916591 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" event={"ID":"64369252-889a-4116-be41-95bc8d95a4e7","Type":"ContainerStarted","Data":"394b71445c7464b2c6f2433ebb46de656a652dc4288b5021fcb6ce97ec3c27ba"} Jul 10 17:12:18.916690 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.916655 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" event={"ID":"64369252-889a-4116-be41-95bc8d95a4e7","Type":"ContainerStarted","Data":"529fadde653674bf5132ae29c7b8ba60aef375b5f5e80d7a3d4b9cbbe4952ee1"} Jul 10 17:12:18.916879 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.916784 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:18.935111 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:18.935048 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" podStartSLOduration=1.544392081 podStartE2EDuration="1.935027633s" totalImagesPullingTime="390.635552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:18.93472045 +0000 UTC m=+2128.610782248" watchObservedRunningTime="2026-07-10 17:12:18.935027633 +0000 UTC m=+2128.611089418" Jul 10 17:12:24.926932 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:24.926905 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:12:49.429785 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.429695 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq"] Jul 10 17:12:49.433113 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.433089 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:49.436388 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.436367 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:12:49.441543 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.441519 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq"] Jul 10 17:12:49.460691 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.460664 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af50a058-m5qzq\" (UID: \"92c98935-6ce2-4c6a-bd7b-066a2ce4471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:49.561173 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.561138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af50a058-m5qzq\" (UID: \"92c98935-6ce2-4c6a-bd7b-066a2ce4471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:49.561799 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.561781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-af50a058-m5qzq\" (UID: \"92c98935-6ce2-4c6a-bd7b-066a2ce4471c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:49.744511 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.744427 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:49.885843 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:49.885760 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq"] Jul 10 17:12:49.888145 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:12:49.888111 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92c98935_6ce2_4c6a_bd7b_066a2ce4471c.slice/crio-2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c WatchSource:0}: Error finding container 2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c: Status 404 returned error can't find the container with id 2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c Jul 10 17:12:50.059896 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:50.059863 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" event={"ID":"92c98935-6ce2-4c6a-bd7b-066a2ce4471c","Type":"ContainerStarted","Data":"c00c9d47afb5e1339526eb036f056d9c27f7da0771befceee1377d0a2cbe2267"} Jul 10 17:12:50.060080 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:50.059921 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" event={"ID":"92c98935-6ce2-4c6a-bd7b-066a2ce4471c","Type":"ContainerStarted","Data":"2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c"} Jul 10 17:12:50.080181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:50.080128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" podStartSLOduration=1.080110493 podStartE2EDuration="1.080110493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:12:50.078785408 +0000 UTC m=+2159.754847186" watchObservedRunningTime="2026-07-10 17:12:50.080110493 +0000 UTC m=+2159.756172278" Jul 10 17:12:51.065043 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:51.065004 2568 generic.go:356] "Generic (PLEG): container finished" podID="92c98935-6ce2-4c6a-bd7b-066a2ce4471c" containerID="c00c9d47afb5e1339526eb036f056d9c27f7da0771befceee1377d0a2cbe2267" exitCode=0 Jul 10 17:12:51.065432 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:51.065070 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" event={"ID":"92c98935-6ce2-4c6a-bd7b-066a2ce4471c","Type":"ContainerDied","Data":"c00c9d47afb5e1339526eb036f056d9c27f7da0771befceee1377d0a2cbe2267"} Jul 10 17:12:52.201592 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:52.201567 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:52.284164 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:52.284122 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca\") pod \"92c98935-6ce2-4c6a-bd7b-066a2ce4471c\" (UID: \"92c98935-6ce2-4c6a-bd7b-066a2ce4471c\") " Jul 10 17:12:52.284523 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:52.284496 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92c98935-6ce2-4c6a-bd7b-066a2ce4471c" (UID: "92c98935-6ce2-4c6a-bd7b-066a2ce4471c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:12:52.385651 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:52.385549 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92c98935-6ce2-4c6a-bd7b-066a2ce4471c-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:12:53.076137 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:53.076108 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" event={"ID":"92c98935-6ce2-4c6a-bd7b-066a2ce4471c","Type":"ContainerDied","Data":"2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c"} Jul 10 17:12:53.076294 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:53.076148 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-af50a058-m5qzq" Jul 10 17:12:53.076294 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:53.076159 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e00a271c7de4156c047c3acfed07f0f73075f0444f4a4d6ab5384e0cf218e9c" Jul 10 17:12:59.948686 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:59.948655 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:12:59.949160 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:12:59.948875 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" podUID="64369252-889a-4116-be41-95bc8d95a4e7" containerName="maas-api" containerID="cri-o://394b71445c7464b2c6f2433ebb46de656a652dc4288b5021fcb6ce97ec3c27ba" gracePeriod=30 Jul 10 17:13:00.111455 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.111421 2568 generic.go:356] "Generic (PLEG): container finished" podID="64369252-889a-4116-be41-95bc8d95a4e7" containerID="394b71445c7464b2c6f2433ebb46de656a652dc4288b5021fcb6ce97ec3c27ba" exitCode=0 Jul 10 17:13:00.111610 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.111493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" event={"ID":"64369252-889a-4116-be41-95bc8d95a4e7","Type":"ContainerDied","Data":"394b71445c7464b2c6f2433ebb46de656a652dc4288b5021fcb6ce97ec3c27ba"} Jul 10 17:13:00.199285 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.199225 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:13:00.249639 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.249600 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca\") pod \"64369252-889a-4116-be41-95bc8d95a4e7\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " Jul 10 17:13:00.249808 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.249675 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hhgb\" (UniqueName: \"kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb\") pod \"64369252-889a-4116-be41-95bc8d95a4e7\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " Jul 10 17:13:00.249808 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.249713 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") pod \"64369252-889a-4116-be41-95bc8d95a4e7\" (UID: \"64369252-889a-4116-be41-95bc8d95a4e7\") " Jul 10 17:13:00.250023 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.250000 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "64369252-889a-4116-be41-95bc8d95a4e7" (UID: "64369252-889a-4116-be41-95bc8d95a4e7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:13:00.251614 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.251572 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb" (OuterVolumeSpecName: "kube-api-access-5hhgb") pod "64369252-889a-4116-be41-95bc8d95a4e7" (UID: "64369252-889a-4116-be41-95bc8d95a4e7"). InnerVolumeSpecName "kube-api-access-5hhgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:13:00.251757 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.251671 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "64369252-889a-4116-be41-95bc8d95a4e7" (UID: "64369252-889a-4116-be41-95bc8d95a4e7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:13:00.350656 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.350625 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hhgb\" (UniqueName: \"kubernetes.io/projected/64369252-889a-4116-be41-95bc8d95a4e7-kube-api-access-5hhgb\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:13:00.350849 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.350676 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/64369252-889a-4116-be41-95bc8d95a4e7-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:13:00.350849 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:00.350686 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/64369252-889a-4116-be41-95bc8d95a4e7-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:13:01.116989 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:01.116959 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" Jul 10 17:13:01.117455 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:01.116979 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2" event={"ID":"64369252-889a-4116-be41-95bc8d95a4e7","Type":"ContainerDied","Data":"529fadde653674bf5132ae29c7b8ba60aef375b5f5e80d7a3d4b9cbbe4952ee1"} Jul 10 17:13:01.117455 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:01.117058 2568 scope.go:117] "RemoveContainer" containerID="394b71445c7464b2c6f2433ebb46de656a652dc4288b5021fcb6ce97ec3c27ba" Jul 10 17:13:01.135242 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:01.135214 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:13:01.142058 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:01.142034 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-af50a058-86cf44c85b-7vxb2"] Jul 10 17:13:03.043640 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:03.043608 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64369252-889a-4116-be41-95bc8d95a4e7" path="/var/lib/kubelet/pods/64369252-889a-4116-be41-95bc8d95a4e7/volumes" Jul 10 17:13:51.057671 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:13:51.057642 2568 scope.go:117] "RemoveContainer" containerID="8b35b865dfee52e70f90f0233745b69620bbc291a5ff738e4ff5400a3e9969e0" Jul 10 17:14:51.080101 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:14:51.080071 2568 scope.go:117] "RemoveContainer" containerID="28665ab0a53da96cb6838a05cc05c2b8eb572678bd99deefdd9722ab2e03c127" Jul 10 17:15:00.168454 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168423 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7"] Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168725 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64369252-889a-4116-be41-95bc8d95a4e7" containerName="maas-api" Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168735 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64369252-889a-4116-be41-95bc8d95a4e7" containerName="maas-api" Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168759 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92c98935-6ce2-4c6a-bd7b-066a2ce4471c" containerName="revoke-keys" Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168765 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="92c98935-6ce2-4c6a-bd7b-066a2ce4471c" containerName="revoke-keys" Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168820 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64369252-889a-4116-be41-95bc8d95a4e7" containerName="maas-api" Jul 10 17:15:00.168972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.168827 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="92c98935-6ce2-4c6a-bd7b-066a2ce4471c" containerName="revoke-keys" Jul 10 17:15:00.171625 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.171609 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" Jul 10 17:15:00.174487 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.174461 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:15:00.178365 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.178329 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7"] Jul 10 17:15:00.182605 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.182582 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" Jul 10 17:15:00.314355 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.314328 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7"] Jul 10 17:15:00.316158 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:15:00.316125 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab27482_63c8_478f_a934_b2540649241c.slice/crio-fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea WatchSource:0}: Error finding container fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea: Status 404 returned error can't find the container with id fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea Jul 10 17:15:00.656156 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.656119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" event={"ID":"4ab27482-63c8-478f-a934-b2540649241c","Type":"ContainerStarted","Data":"d1083148e4cedce4588571f10ed5ed29fc60ff29e75d118041d69caf0d08a470"} Jul 10 17:15:00.656339 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.656169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" event={"ID":"4ab27482-63c8-478f-a934-b2540649241c","Type":"ContainerStarted","Data":"fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea"} Jul 10 17:15:00.674676 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:00.674625 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" podStartSLOduration=0.674609954 podStartE2EDuration="674.609954ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:00.673288963 +0000 UTC m=+2290.349350748" watchObservedRunningTime="2026-07-10 17:15:00.674609954 +0000 UTC m=+2290.350671761" Jul 10 17:15:01.662134 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:01.662047 2568 generic.go:356] "Generic (PLEG): container finished" podID="4ab27482-63c8-478f-a934-b2540649241c" containerID="d1083148e4cedce4588571f10ed5ed29fc60ff29e75d118041d69caf0d08a470" exitCode=0 Jul 10 17:15:01.662470 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:01.662136 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" event={"ID":"4ab27482-63c8-478f-a934-b2540649241c","Type":"ContainerDied","Data":"d1083148e4cedce4588571f10ed5ed29fc60ff29e75d118041d69caf0d08a470"} Jul 10 17:15:02.795467 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:02.795444 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" Jul 10 17:15:03.677194 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:03.677148 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" event={"ID":"4ab27482-63c8-478f-a934-b2540649241c","Type":"ContainerDied","Data":"fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea"} Jul 10 17:15:03.677378 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:03.677224 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29728395-pckp7" Jul 10 17:15:03.677378 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:03.677278 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde42973f6da78445163651bedf3dfe8cae8bd4ee4a08726e7aaaaba157068ea" Jul 10 17:15:14.016114 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.016084 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:15:14.016548 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.016390 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab27482-63c8-478f-a934-b2540649241c" containerName="cleanup" Jul 10 17:15:14.016548 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.016400 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab27482-63c8-478f-a934-b2540649241c" containerName="cleanup" Jul 10 17:15:14.016548 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.016461 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab27482-63c8-478f-a934-b2540649241c" containerName="cleanup" Jul 10 17:15:14.019621 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.019606 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.022520 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.022494 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:15:14.022669 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.022555 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-889e72-serving-cert\"" Jul 10 17:15:14.022669 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.022632 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:15:14.023821 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.023796 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:15:14.030664 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.030643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:15:14.119421 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.119389 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.119577 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.119428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.119577 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.119495 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwcx\" (UniqueName: \"kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.220010 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.219972 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.220205 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.220017 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.220205 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.220053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwcx\" (UniqueName: \"kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.220205 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:15:14.220121 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-serving-cert: secret "maas-api-e2e-inf-a-889e72-serving-cert" not found Jul 10 17:15:14.220205 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:15:14.220205 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls podName:aa068148-3142-48d8-a871-6af9dfb7a697 nodeName:}" failed. No retries permitted until 2026-07-10 17:15:14.720180495 +0000 UTC m=+2304.396242257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls") pod "maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" (UID: "aa068148-3142-48d8-a871-6af9dfb7a697") : secret "maas-api-e2e-inf-a-889e72-serving-cert" not found Jul 10 17:15:14.220762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.220719 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.229825 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.229799 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwcx\" (UniqueName: \"kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.724513 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.724472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.726913 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.726891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") pod \"maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:14.930129 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:14.930099 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:15.064864 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:15.064761 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:15:15.067242 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:15:15.067217 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa068148_3142_48d8_a871_6af9dfb7a697.slice/crio-3af838b6beaf518809d949e4932e85d43a84f91e1f4df0b87cb899cdffd7892d WatchSource:0}: Error finding container 3af838b6beaf518809d949e4932e85d43a84f91e1f4df0b87cb899cdffd7892d: Status 404 returned error can't find the container with id 3af838b6beaf518809d949e4932e85d43a84f91e1f4df0b87cb899cdffd7892d Jul 10 17:15:15.739451 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:15.739411 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" event={"ID":"aa068148-3142-48d8-a871-6af9dfb7a697","Type":"ContainerStarted","Data":"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c"} Jul 10 17:15:15.739633 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:15.739518 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:15.739633 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:15.739543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" event={"ID":"aa068148-3142-48d8-a871-6af9dfb7a697","Type":"ContainerStarted","Data":"3af838b6beaf518809d949e4932e85d43a84f91e1f4df0b87cb899cdffd7892d"} Jul 10 17:15:15.759598 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:15.759551 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" podStartSLOduration=2.291522975 podStartE2EDuration="2.759535964s" totalImagesPullingTime="468.012989ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:15.758192639 +0000 UTC m=+2305.434254424" watchObservedRunningTime="2026-07-10 17:15:15.759535964 +0000 UTC m=+2305.435597765" Jul 10 17:15:21.750939 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:21.750908 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:15:22.088400 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.088326 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:15:22.091889 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.091868 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.095054 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.095034 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-a0f0a2-serving-cert\"" Jul 10 17:15:22.104507 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.104481 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:15:22.188461 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.188424 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.188624 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.188486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bz74\" (UniqueName: \"kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.188624 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.188534 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.289023 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.288988 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bz74\" (UniqueName: \"kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.289188 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.289046 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.289188 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.289095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.289261 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:15:22.289207 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-serving-cert: secret "maas-api-e2e-inf-b-a0f0a2-serving-cert" not found Jul 10 17:15:22.289298 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:15:22.289267 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls podName:74e40a01-a121-4e6e-afe5-22bc92d2db85 nodeName:}" failed. No retries permitted until 2026-07-10 17:15:22.789251706 +0000 UTC m=+2312.465313473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls") pod "maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" (UID: "74e40a01-a121-4e6e-afe5-22bc92d2db85") : secret "maas-api-e2e-inf-b-a0f0a2-serving-cert" not found Jul 10 17:15:22.289717 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.289699 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.299199 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.299168 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bz74\" (UniqueName: \"kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.793325 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.793296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:22.795959 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:22.795932 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") pod \"maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:23.001336 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.001301 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:23.137152 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.137115 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:15:23.140121 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:15:23.140091 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e40a01_a121_4e6e_afe5_22bc92d2db85.slice/crio-3defd34dc3ace4ffbbed268fb605e0b2f8bfc0a3c57ec5667e9a9e7724d8347a WatchSource:0}: Error finding container 3defd34dc3ace4ffbbed268fb605e0b2f8bfc0a3c57ec5667e9a9e7724d8347a: Status 404 returned error can't find the container with id 3defd34dc3ace4ffbbed268fb605e0b2f8bfc0a3c57ec5667e9a9e7724d8347a Jul 10 17:15:23.776231 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.776196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" event={"ID":"74e40a01-a121-4e6e-afe5-22bc92d2db85","Type":"ContainerStarted","Data":"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c"} Jul 10 17:15:23.776388 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.776247 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" event={"ID":"74e40a01-a121-4e6e-afe5-22bc92d2db85","Type":"ContainerStarted","Data":"3defd34dc3ace4ffbbed268fb605e0b2f8bfc0a3c57ec5667e9a9e7724d8347a"} Jul 10 17:15:23.776388 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.776327 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:15:23.794964 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:23.794916 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" podStartSLOduration=1.376910612 podStartE2EDuration="1.794901263s" totalImagesPullingTime="417.990651ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:15:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:15:23.792357687 +0000 UTC m=+2313.468419483" watchObservedRunningTime="2026-07-10 17:15:23.794901263 +0000 UTC m=+2313.470963049" Jul 10 17:15:29.787903 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:15:29.787872 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:16:20.816472 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:20.816436 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz"] Jul 10 17:16:20.819862 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:20.819843 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:20.823327 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:20.823306 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:16:20.834434 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:20.834404 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz"] Jul 10 17:16:20.906498 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:20.906464 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz\" (UID: \"beb1c8b2-722c-45c9-911e-2e3250c3f7a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:21.007599 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:21.007567 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz\" (UID: \"beb1c8b2-722c-45c9-911e-2e3250c3f7a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:21.008235 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:21.008216 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz\" (UID: \"beb1c8b2-722c-45c9-911e-2e3250c3f7a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:21.129676 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:21.129583 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:21.259879 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:21.259851 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz"] Jul 10 17:16:21.262047 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:16:21.262015 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb1c8b2_722c_45c9_911e_2e3250c3f7a5.slice/crio-06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a WatchSource:0}: Error finding container 06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a: Status 404 returned error can't find the container with id 06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a Jul 10 17:16:22.053590 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:22.053553 2568 generic.go:356] "Generic (PLEG): container finished" podID="beb1c8b2-722c-45c9-911e-2e3250c3f7a5" containerID="19f351fb02b1a4d5d13d6c988c5df374120f577f027455b53486172645b6131b" exitCode=0 Jul 10 17:16:22.053999 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:22.053633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" event={"ID":"beb1c8b2-722c-45c9-911e-2e3250c3f7a5","Type":"ContainerDied","Data":"19f351fb02b1a4d5d13d6c988c5df374120f577f027455b53486172645b6131b"} Jul 10 17:16:22.053999 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:22.053685 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" event={"ID":"beb1c8b2-722c-45c9-911e-2e3250c3f7a5","Type":"ContainerStarted","Data":"06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a"} Jul 10 17:16:23.184116 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:23.184088 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:23.331382 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:23.331296 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca\") pod \"beb1c8b2-722c-45c9-911e-2e3250c3f7a5\" (UID: \"beb1c8b2-722c-45c9-911e-2e3250c3f7a5\") " Jul 10 17:16:23.331698 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:23.331676 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "beb1c8b2-722c-45c9-911e-2e3250c3f7a5" (UID: "beb1c8b2-722c-45c9-911e-2e3250c3f7a5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:23.432756 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:23.432706 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beb1c8b2-722c-45c9-911e-2e3250c3f7a5-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:24.067238 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:24.067209 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" Jul 10 17:16:24.067414 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:24.067208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz" event={"ID":"beb1c8b2-722c-45c9-911e-2e3250c3f7a5","Type":"ContainerDied","Data":"06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a"} Jul 10 17:16:24.067414 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:24.067340 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06523fc520ad5297103630372c8fde15c74a0342897d261671a5eb678951477a" Jul 10 17:16:31.257669 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.257628 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:16:31.258245 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.257932 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" podUID="aa068148-3142-48d8-a871-6af9dfb7a697" containerName="maas-api" containerID="cri-o://b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c" gracePeriod=30 Jul 10 17:16:31.507813 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.507720 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:16:31.606583 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.606548 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca\") pod \"aa068148-3142-48d8-a871-6af9dfb7a697\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " Jul 10 17:16:31.606797 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.606656 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") pod \"aa068148-3142-48d8-a871-6af9dfb7a697\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " Jul 10 17:16:31.606797 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.606686 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbwcx\" (UniqueName: \"kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx\") pod \"aa068148-3142-48d8-a871-6af9dfb7a697\" (UID: \"aa068148-3142-48d8-a871-6af9dfb7a697\") " Jul 10 17:16:31.607046 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.607020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aa068148-3142-48d8-a871-6af9dfb7a697" (UID: "aa068148-3142-48d8-a871-6af9dfb7a697"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:31.608763 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.608728 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx" (OuterVolumeSpecName: "kube-api-access-hbwcx") pod "aa068148-3142-48d8-a871-6af9dfb7a697" (UID: "aa068148-3142-48d8-a871-6af9dfb7a697"). InnerVolumeSpecName "kube-api-access-hbwcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:31.608832 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.608792 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aa068148-3142-48d8-a871-6af9dfb7a697" (UID: "aa068148-3142-48d8-a871-6af9dfb7a697"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:31.707502 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.707472 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwcx\" (UniqueName: \"kubernetes.io/projected/aa068148-3142-48d8-a871-6af9dfb7a697-kube-api-access-hbwcx\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:31.707502 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.707499 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa068148-3142-48d8-a871-6af9dfb7a697-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:31.707502 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:31.707509 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aa068148-3142-48d8-a871-6af9dfb7a697-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:32.105384 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.105349 2568 generic.go:356] "Generic (PLEG): container finished" podID="aa068148-3142-48d8-a871-6af9dfb7a697" containerID="b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c" exitCode=0 Jul 10 17:16:32.105554 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.105399 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" Jul 10 17:16:32.105554 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.105430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" event={"ID":"aa068148-3142-48d8-a871-6af9dfb7a697","Type":"ContainerDied","Data":"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c"} Jul 10 17:16:32.105554 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.105493 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb" event={"ID":"aa068148-3142-48d8-a871-6af9dfb7a697","Type":"ContainerDied","Data":"3af838b6beaf518809d949e4932e85d43a84f91e1f4df0b87cb899cdffd7892d"} Jul 10 17:16:32.105554 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.105516 2568 scope.go:117] "RemoveContainer" containerID="b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c" Jul 10 17:16:32.114514 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.114499 2568 scope.go:117] "RemoveContainer" containerID="b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c" Jul 10 17:16:32.114808 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:16:32.114780 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c\": container with ID starting with b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c not found: ID does not exist" containerID="b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c" Jul 10 17:16:32.114873 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.114820 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c"} err="failed to get container status \"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c\": rpc error: code = NotFound desc = could not find container \"b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c\": container with ID starting with b217b26afc44ee4bf52c7ce2e20dde288367ebeda3a5cc4e17788046845a860c not found: ID does not exist" Jul 10 17:16:32.135660 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.135630 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:16:32.137428 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:32.137403 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-889e72-7c88686d99-n2wpb"] Jul 10 17:16:33.043505 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:33.043462 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa068148-3142-48d8-a871-6af9dfb7a697" path="/var/lib/kubelet/pods/aa068148-3142-48d8-a871-6af9dfb7a697/volumes" Jul 10 17:16:46.950416 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950380 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf"] Jul 10 17:16:46.950815 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950708 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa068148-3142-48d8-a871-6af9dfb7a697" containerName="maas-api" Jul 10 17:16:46.950815 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950719 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa068148-3142-48d8-a871-6af9dfb7a697" containerName="maas-api" Jul 10 17:16:46.950815 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950735 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beb1c8b2-722c-45c9-911e-2e3250c3f7a5" containerName="revoke-keys" Jul 10 17:16:46.950815 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950757 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="beb1c8b2-722c-45c9-911e-2e3250c3f7a5" containerName="revoke-keys" Jul 10 17:16:46.950815 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950814 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa068148-3142-48d8-a871-6af9dfb7a697" containerName="maas-api" Jul 10 17:16:46.951122 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.950824 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="beb1c8b2-722c-45c9-911e-2e3250c3f7a5" containerName="revoke-keys" Jul 10 17:16:46.955106 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.955092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:46.957899 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.957877 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:16:46.962465 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:46.962440 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf"] Jul 10 17:16:47.026673 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:47.026645 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf\" (UID: \"c27da0ed-b10f-4887-9abd-29b15ee8f5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:47.127510 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:47.127471 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf\" (UID: \"c27da0ed-b10f-4887-9abd-29b15ee8f5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:47.128140 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:47.128120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf\" (UID: \"c27da0ed-b10f-4887-9abd-29b15ee8f5d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:47.266623 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:47.266534 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:47.399411 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:47.399387 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf"] Jul 10 17:16:47.401996 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:16:47.401970 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27da0ed_b10f_4887_9abd_29b15ee8f5d9.slice/crio-4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b WatchSource:0}: Error finding container 4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b: Status 404 returned error can't find the container with id 4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b Jul 10 17:16:48.184142 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:48.184101 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" event={"ID":"c27da0ed-b10f-4887-9abd-29b15ee8f5d9","Type":"ContainerStarted","Data":"2e290ffcdef28328d85239ba8b58119c226706d6776b770198b4fe45855e9ab7"} Jul 10 17:16:48.184540 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:48.184163 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" event={"ID":"c27da0ed-b10f-4887-9abd-29b15ee8f5d9","Type":"ContainerStarted","Data":"4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b"} Jul 10 17:16:48.201011 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:48.200963 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" podStartSLOduration=2.200949262 podStartE2EDuration="2.200949262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:16:48.198771918 +0000 UTC m=+2397.874833701" watchObservedRunningTime="2026-07-10 17:16:48.200949262 +0000 UTC m=+2397.877011046" Jul 10 17:16:49.190193 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:49.190160 2568 generic.go:356] "Generic (PLEG): container finished" podID="c27da0ed-b10f-4887-9abd-29b15ee8f5d9" containerID="2e290ffcdef28328d85239ba8b58119c226706d6776b770198b4fe45855e9ab7" exitCode=0 Jul 10 17:16:49.190553 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:49.190202 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" event={"ID":"c27da0ed-b10f-4887-9abd-29b15ee8f5d9","Type":"ContainerDied","Data":"2e290ffcdef28328d85239ba8b58119c226706d6776b770198b4fe45855e9ab7"} Jul 10 17:16:50.328630 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.328608 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:50.356240 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.356214 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca\") pod \"c27da0ed-b10f-4887-9abd-29b15ee8f5d9\" (UID: \"c27da0ed-b10f-4887-9abd-29b15ee8f5d9\") " Jul 10 17:16:50.356639 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.356614 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c27da0ed-b10f-4887-9abd-29b15ee8f5d9" (UID: "c27da0ed-b10f-4887-9abd-29b15ee8f5d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:50.457325 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.457243 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27da0ed-b10f-4887-9abd-29b15ee8f5d9-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:50.989232 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.989203 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:16:50.993954 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:50.993934 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:16:51.199928 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:51.199901 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" Jul 10 17:16:51.199928 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:51.199917 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf" event={"ID":"c27da0ed-b10f-4887-9abd-29b15ee8f5d9","Type":"ContainerDied","Data":"4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b"} Jul 10 17:16:51.200154 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:51.199958 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e80877ca377db33bf295616728e1f0f027ecf689af83e84b4a232ad9d2ad74b" Jul 10 17:16:57.364130 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.364097 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:16:57.364590 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.364353 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" podUID="74e40a01-a121-4e6e-afe5-22bc92d2db85" containerName="maas-api" containerID="cri-o://dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c" gracePeriod=30 Jul 10 17:16:57.805824 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.805800 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:16:57.919194 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.919109 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bz74\" (UniqueName: \"kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74\") pod \"74e40a01-a121-4e6e-afe5-22bc92d2db85\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " Jul 10 17:16:57.919194 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.919141 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") pod \"74e40a01-a121-4e6e-afe5-22bc92d2db85\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " Jul 10 17:16:57.919421 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.919345 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca\") pod \"74e40a01-a121-4e6e-afe5-22bc92d2db85\" (UID: \"74e40a01-a121-4e6e-afe5-22bc92d2db85\") " Jul 10 17:16:57.919724 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.919698 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "74e40a01-a121-4e6e-afe5-22bc92d2db85" (UID: "74e40a01-a121-4e6e-afe5-22bc92d2db85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:16:57.921183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.921161 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74" (OuterVolumeSpecName: "kube-api-access-5bz74") pod "74e40a01-a121-4e6e-afe5-22bc92d2db85" (UID: "74e40a01-a121-4e6e-afe5-22bc92d2db85"). InnerVolumeSpecName "kube-api-access-5bz74". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:16:57.921268 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:57.921191 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "74e40a01-a121-4e6e-afe5-22bc92d2db85" (UID: "74e40a01-a121-4e6e-afe5-22bc92d2db85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:16:58.020937 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.020895 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74e40a01-a121-4e6e-afe5-22bc92d2db85-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:58.020937 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.020930 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bz74\" (UniqueName: \"kubernetes.io/projected/74e40a01-a121-4e6e-afe5-22bc92d2db85-kube-api-access-5bz74\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:58.020937 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.020944 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74e40a01-a121-4e6e-afe5-22bc92d2db85-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:16:58.234034 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.233990 2568 generic.go:356] "Generic (PLEG): container finished" podID="74e40a01-a121-4e6e-afe5-22bc92d2db85" containerID="dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c" exitCode=0 Jul 10 17:16:58.234034 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.234033 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" Jul 10 17:16:58.234279 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.234073 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" event={"ID":"74e40a01-a121-4e6e-afe5-22bc92d2db85","Type":"ContainerDied","Data":"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c"} Jul 10 17:16:58.234279 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.234125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn" event={"ID":"74e40a01-a121-4e6e-afe5-22bc92d2db85","Type":"ContainerDied","Data":"3defd34dc3ace4ffbbed268fb605e0b2f8bfc0a3c57ec5667e9a9e7724d8347a"} Jul 10 17:16:58.234279 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.234147 2568 scope.go:117] "RemoveContainer" containerID="dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c" Jul 10 17:16:58.243195 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.243177 2568 scope.go:117] "RemoveContainer" containerID="dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c" Jul 10 17:16:58.243473 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:16:58.243456 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c\": container with ID starting with dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c not found: ID does not exist" containerID="dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c" Jul 10 17:16:58.243622 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.243480 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c"} err="failed to get container status \"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c\": rpc error: code = NotFound desc = could not find container \"dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c\": container with ID starting with dac0eccc854ca8c517ca8c72b8dfbc0005d59729795896c4a509f545d807633c not found: ID does not exist" Jul 10 17:16:58.256364 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.256338 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:16:58.260885 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:58.260865 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-a0f0a2-856b54f4c9-4x9dn"] Jul 10 17:16:59.042629 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:16:59.042595 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e40a01-a121-4e6e-afe5-22bc92d2db85" path="/var/lib/kubelet/pods/74e40a01-a121-4e6e-afe5-22bc92d2db85/volumes" Jul 10 17:17:16.894612 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.894527 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.894908 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27da0ed-b10f-4887-9abd-29b15ee8f5d9" containerName="revoke-keys" Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.894920 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27da0ed-b10f-4887-9abd-29b15ee8f5d9" containerName="revoke-keys" Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.894939 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e40a01-a121-4e6e-afe5-22bc92d2db85" containerName="maas-api" Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.894945 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e40a01-a121-4e6e-afe5-22bc92d2db85" containerName="maas-api" Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.895002 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e40a01-a121-4e6e-afe5-22bc92d2db85" containerName="maas-api" Jul 10 17:17:16.895181 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.895011 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27da0ed-b10f-4887-9abd-29b15ee8f5d9" containerName="revoke-keys" Jul 10 17:17:16.899372 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.899348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:16.902414 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.902393 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:17:16.902665 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.902649 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5e0f6f-serving-cert\"" Jul 10 17:17:16.904005 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.903986 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:17:16.904103 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.904025 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:17:16.912242 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.912221 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:16.976874 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.976833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:16.977065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.976944 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:16.977065 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:16.976975 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzbm7\" (UniqueName: \"kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.078114 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.078082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzbm7\" (UniqueName: \"kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.078281 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.078123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.078281 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.078185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.078354 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:17:17.078287 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-serving-cert: secret "maas-api-e2e-api-a-5e0f6f-serving-cert" not found Jul 10 17:17:17.078388 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:17:17.078352 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls podName:bd46b967-d37a-42e0-b8fd-902396a38e94 nodeName:}" failed. No retries permitted until 2026-07-10 17:17:17.578332486 +0000 UTC m=+2427.254394248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls") pod "maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" (UID: "bd46b967-d37a-42e0-b8fd-902396a38e94") : secret "maas-api-e2e-api-a-5e0f6f-serving-cert" not found Jul 10 17:17:17.078877 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.078859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.091052 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.091031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzbm7\" (UniqueName: \"kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.581832 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.581791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.584320 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.584296 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") pod \"maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.809762 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.809715 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:17.939932 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.939906 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:17.941525 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:17:17.941497 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd46b967_d37a_42e0_b8fd_902396a38e94.slice/crio-f8185b28dbe3cdfd9a96d0214cfd90f17079d1441b8408ed5fa2a324efa2b70e WatchSource:0}: Error finding container f8185b28dbe3cdfd9a96d0214cfd90f17079d1441b8408ed5fa2a324efa2b70e: Status 404 returned error can't find the container with id f8185b28dbe3cdfd9a96d0214cfd90f17079d1441b8408ed5fa2a324efa2b70e Jul 10 17:17:17.942898 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:17.942880 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 10 17:17:18.328438 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:18.328396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" event={"ID":"bd46b967-d37a-42e0-b8fd-902396a38e94","Type":"ContainerStarted","Data":"f8185b28dbe3cdfd9a96d0214cfd90f17079d1441b8408ed5fa2a324efa2b70e"} Jul 10 17:17:19.334486 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:19.334455 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" event={"ID":"bd46b967-d37a-42e0-b8fd-902396a38e94","Type":"ContainerStarted","Data":"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8"} Jul 10 17:17:19.334884 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:19.334567 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:19.353398 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:19.353347 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" podStartSLOduration=2.462471589 podStartE2EDuration="3.353325035s" totalImagesPullingTime="890.853446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:19.351849124 +0000 UTC m=+2429.027910921" watchObservedRunningTime="2026-07-10 17:17:19.353325035 +0000 UTC m=+2429.029386820" Jul 10 17:17:24.836241 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.836206 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:17:24.839831 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.839802 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:24.842550 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.842528 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-dd88b4-serving-cert\"" Jul 10 17:17:24.852510 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.852484 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:17:24.947139 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.947107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:24.947341 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.947144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mp5m\" (UniqueName: \"kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:24.947341 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:24.947169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.047664 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.047628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.047664 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.047666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mp5m\" (UniqueName: \"kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.047890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.047697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.047933 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:17:25.047886 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-serving-cert: secret "maas-api-e2e-api-b-dd88b4-serving-cert" not found Jul 10 17:17:25.047975 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:17:25.047967 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls podName:9ba03504-b695-40f4-91cb-5373539b91ee nodeName:}" failed. No retries permitted until 2026-07-10 17:17:25.5479468 +0000 UTC m=+2435.224008567 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls") pod "maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" (UID: "9ba03504-b695-40f4-91cb-5373539b91ee") : secret "maas-api-e2e-api-b-dd88b4-serving-cert" not found Jul 10 17:17:25.048351 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.048331 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.057292 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.057262 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mp5m\" (UniqueName: \"kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.344943 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.344916 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:25.551138 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.551099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.553529 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.553509 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") pod \"maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.750471 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.750428 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:25.885498 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:25.885472 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:17:25.888225 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:17:25.888198 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba03504_b695_40f4_91cb_5373539b91ee.slice/crio-c4bf80bfd8b470f2ba9183a3dade3da2ca8d51454a537ca8bc0d26baad892490 WatchSource:0}: Error finding container c4bf80bfd8b470f2ba9183a3dade3da2ca8d51454a537ca8bc0d26baad892490: Status 404 returned error can't find the container with id c4bf80bfd8b470f2ba9183a3dade3da2ca8d51454a537ca8bc0d26baad892490 Jul 10 17:17:26.373128 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:26.373097 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" event={"ID":"9ba03504-b695-40f4-91cb-5373539b91ee","Type":"ContainerStarted","Data":"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd"} Jul 10 17:17:26.373291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:26.373148 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" event={"ID":"9ba03504-b695-40f4-91cb-5373539b91ee","Type":"ContainerStarted","Data":"c4bf80bfd8b470f2ba9183a3dade3da2ca8d51454a537ca8bc0d26baad892490"} Jul 10 17:17:26.373291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:26.373233 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:26.400106 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:26.400063 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" podStartSLOduration=2.124560728 podStartE2EDuration="2.400046881s" totalImagesPullingTime="275.486153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:17:26.397689725 +0000 UTC m=+2436.073751503" watchObservedRunningTime="2026-07-10 17:17:26.400046881 +0000 UTC m=+2436.076108717" Jul 10 17:17:32.383273 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:32.383241 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:17:47.685373 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.685341 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:47.685896 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.685677 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" podUID="bd46b967-d37a-42e0-b8fd-902396a38e94" containerName="maas-api" containerID="cri-o://efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8" gracePeriod=30 Jul 10 17:17:47.931844 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.931815 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:47.937287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.937224 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca\") pod \"bd46b967-d37a-42e0-b8fd-902396a38e94\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " Jul 10 17:17:47.937287 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.937266 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzbm7\" (UniqueName: \"kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7\") pod \"bd46b967-d37a-42e0-b8fd-902396a38e94\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " Jul 10 17:17:47.937416 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.937299 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") pod \"bd46b967-d37a-42e0-b8fd-902396a38e94\" (UID: \"bd46b967-d37a-42e0-b8fd-902396a38e94\") " Jul 10 17:17:47.937559 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.937533 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd46b967-d37a-42e0-b8fd-902396a38e94" (UID: "bd46b967-d37a-42e0-b8fd-902396a38e94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:17:47.939297 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.939271 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd46b967-d37a-42e0-b8fd-902396a38e94" (UID: "bd46b967-d37a-42e0-b8fd-902396a38e94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:17:47.939392 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:47.939346 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7" (OuterVolumeSpecName: "kube-api-access-kzbm7") pod "bd46b967-d37a-42e0-b8fd-902396a38e94" (UID: "bd46b967-d37a-42e0-b8fd-902396a38e94"). InnerVolumeSpecName "kube-api-access-kzbm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:17:48.037847 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.037806 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd46b967-d37a-42e0-b8fd-902396a38e94-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:17:48.037847 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.037834 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzbm7\" (UniqueName: \"kubernetes.io/projected/bd46b967-d37a-42e0-b8fd-902396a38e94-kube-api-access-kzbm7\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:17:48.037847 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.037844 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd46b967-d37a-42e0-b8fd-902396a38e94-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:17:48.478584 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.478545 2568 generic.go:356] "Generic (PLEG): container finished" podID="bd46b967-d37a-42e0-b8fd-902396a38e94" containerID="efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8" exitCode=0 Jul 10 17:17:48.478764 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.478591 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" Jul 10 17:17:48.478764 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.478633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" event={"ID":"bd46b967-d37a-42e0-b8fd-902396a38e94","Type":"ContainerDied","Data":"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8"} Jul 10 17:17:48.478764 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.478687 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx" event={"ID":"bd46b967-d37a-42e0-b8fd-902396a38e94","Type":"ContainerDied","Data":"f8185b28dbe3cdfd9a96d0214cfd90f17079d1441b8408ed5fa2a324efa2b70e"} Jul 10 17:17:48.478764 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.478710 2568 scope.go:117] "RemoveContainer" containerID="efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8" Jul 10 17:17:48.487582 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.487556 2568 scope.go:117] "RemoveContainer" containerID="efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8" Jul 10 17:17:48.487869 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:17:48.487849 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8\": container with ID starting with efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8 not found: ID does not exist" containerID="efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8" Jul 10 17:17:48.487930 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.487880 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8"} err="failed to get container status \"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8\": rpc error: code = NotFound desc = could not find container \"efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8\": container with ID starting with efd20e7470c4a5bd1b3b787ad2706afa07f7f2525e0469678af39372faa7dcb8 not found: ID does not exist" Jul 10 17:17:48.502381 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.502354 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:48.506129 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:48.506107 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5e0f6f-6c6c4989b5-fmgxx"] Jul 10 17:17:49.044044 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:17:49.044006 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd46b967-d37a-42e0-b8fd-902396a38e94" path="/var/lib/kubelet/pods/bd46b967-d37a-42e0-b8fd-902396a38e94/volumes" Jul 10 17:18:03.197899 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.197859 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w"] Jul 10 17:18:03.198372 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.198172 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd46b967-d37a-42e0-b8fd-902396a38e94" containerName="maas-api" Jul 10 17:18:03.198372 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.198182 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd46b967-d37a-42e0-b8fd-902396a38e94" containerName="maas-api" Jul 10 17:18:03.198372 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.198261 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd46b967-d37a-42e0-b8fd-902396a38e94" containerName="maas-api" Jul 10 17:18:03.201231 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.201216 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:03.203852 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.203832 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:18:03.209300 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.209276 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w"] Jul 10 17:18:03.250232 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.250198 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w\" (UID: \"6c4e8f8c-b9f0-494f-a914-5d241616590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:03.351529 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.351493 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w\" (UID: \"6c4e8f8c-b9f0-494f-a914-5d241616590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:03.352155 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.352134 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w\" (UID: \"6c4e8f8c-b9f0-494f-a914-5d241616590c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:03.512007 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.511911 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:03.644876 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:03.644850 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w"] Jul 10 17:18:03.647399 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:18:03.647372 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4e8f8c_b9f0_494f_a914_5d241616590c.slice/crio-983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d WatchSource:0}: Error finding container 983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d: Status 404 returned error can't find the container with id 983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d Jul 10 17:18:04.562221 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:04.562187 2568 generic.go:356] "Generic (PLEG): container finished" podID="6c4e8f8c-b9f0-494f-a914-5d241616590c" containerID="baf9c9451941df6125597cbcd75df9dc975259000a24bdcb073e287ebd4f46b9" exitCode=0 Jul 10 17:18:04.562613 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:04.562251 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" event={"ID":"6c4e8f8c-b9f0-494f-a914-5d241616590c","Type":"ContainerDied","Data":"baf9c9451941df6125597cbcd75df9dc975259000a24bdcb073e287ebd4f46b9"} Jul 10 17:18:04.562613 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:04.562303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" event={"ID":"6c4e8f8c-b9f0-494f-a914-5d241616590c","Type":"ContainerStarted","Data":"983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d"} Jul 10 17:18:05.698807 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:05.698781 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:05.768618 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:05.768583 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca\") pod \"6c4e8f8c-b9f0-494f-a914-5d241616590c\" (UID: \"6c4e8f8c-b9f0-494f-a914-5d241616590c\") " Jul 10 17:18:05.768968 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:05.768943 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6c4e8f8c-b9f0-494f-a914-5d241616590c" (UID: "6c4e8f8c-b9f0-494f-a914-5d241616590c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:18:05.869947 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:05.869854 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c4e8f8c-b9f0-494f-a914-5d241616590c-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:18:06.573052 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:06.573020 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" Jul 10 17:18:06.573225 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:06.573051 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w" event={"ID":"6c4e8f8c-b9f0-494f-a914-5d241616590c","Type":"ContainerDied","Data":"983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d"} Jul 10 17:18:06.573225 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:06.573101 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="983b7ecfbaf9c99a4d45a06972d0dc41a647967eced5ee933f126cd43a196b4d" Jul 10 17:18:13.615672 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.615631 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:18:13.616350 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.615846 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" podUID="9ba03504-b695-40f4-91cb-5373539b91ee" containerName="maas-api" containerID="cri-o://e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd" gracePeriod=30 Jul 10 17:18:13.863796 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.863766 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:18:13.931259 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.931226 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca\") pod \"9ba03504-b695-40f4-91cb-5373539b91ee\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " Jul 10 17:18:13.931516 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.931498 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") pod \"9ba03504-b695-40f4-91cb-5373539b91ee\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " Jul 10 17:18:13.931658 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.931642 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mp5m\" (UniqueName: \"kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m\") pod \"9ba03504-b695-40f4-91cb-5373539b91ee\" (UID: \"9ba03504-b695-40f4-91cb-5373539b91ee\") " Jul 10 17:18:13.931853 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.931686 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9ba03504-b695-40f4-91cb-5373539b91ee" (UID: "9ba03504-b695-40f4-91cb-5373539b91ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:18:13.932158 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.932138 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ba03504-b695-40f4-91cb-5373539b91ee-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:18:13.934130 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.934101 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9ba03504-b695-40f4-91cb-5373539b91ee" (UID: "9ba03504-b695-40f4-91cb-5373539b91ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:18:13.934250 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:13.934152 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m" (OuterVolumeSpecName: "kube-api-access-7mp5m") pod "9ba03504-b695-40f4-91cb-5373539b91ee" (UID: "9ba03504-b695-40f4-91cb-5373539b91ee"). InnerVolumeSpecName "kube-api-access-7mp5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:18:14.033622 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.033573 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9ba03504-b695-40f4-91cb-5373539b91ee-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:18:14.033622 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.033614 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mp5m\" (UniqueName: \"kubernetes.io/projected/9ba03504-b695-40f4-91cb-5373539b91ee-kube-api-access-7mp5m\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:18:14.615111 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.615075 2568 generic.go:356] "Generic (PLEG): container finished" podID="9ba03504-b695-40f4-91cb-5373539b91ee" containerID="e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd" exitCode=0 Jul 10 17:18:14.615318 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.615131 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" Jul 10 17:18:14.615318 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.615140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" event={"ID":"9ba03504-b695-40f4-91cb-5373539b91ee","Type":"ContainerDied","Data":"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd"} Jul 10 17:18:14.615318 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.615184 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh" event={"ID":"9ba03504-b695-40f4-91cb-5373539b91ee","Type":"ContainerDied","Data":"c4bf80bfd8b470f2ba9183a3dade3da2ca8d51454a537ca8bc0d26baad892490"} Jul 10 17:18:14.615318 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.615204 2568 scope.go:117] "RemoveContainer" containerID="e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd" Jul 10 17:18:14.624484 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.624338 2568 scope.go:117] "RemoveContainer" containerID="e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd" Jul 10 17:18:14.624749 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:18:14.624634 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd\": container with ID starting with e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd not found: ID does not exist" containerID="e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd" Jul 10 17:18:14.624749 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.624663 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd"} err="failed to get container status \"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd\": rpc error: code = NotFound desc = could not find container \"e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd\": container with ID starting with e214be50d1ca639c6c77ea91e9adf77e847388a3b69edbad175d1203844776fd not found: ID does not exist" Jul 10 17:18:14.637158 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.637121 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:18:14.643291 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:14.643259 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-dd88b4-d87f45cf8-lhgkh"] Jul 10 17:18:15.043306 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:15.043272 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba03504-b695-40f4-91cb-5373539b91ee" path="/var/lib/kubelet/pods/9ba03504-b695-40f4-91cb-5373539b91ee/volumes" Jul 10 17:18:33.257377 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257338 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257708 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c4e8f8c-b9f0-494f-a914-5d241616590c" containerName="revoke-keys" Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257722 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c4e8f8c-b9f0-494f-a914-5d241616590c" containerName="revoke-keys" Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257730 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba03504-b695-40f4-91cb-5373539b91ee" containerName="maas-api" Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257752 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba03504-b695-40f4-91cb-5373539b91ee" containerName="maas-api" Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257830 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba03504-b695-40f4-91cb-5373539b91ee" containerName="maas-api" Jul 10 17:18:33.257890 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.257847 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c4e8f8c-b9f0-494f-a914-5d241616590c" containerName="revoke-keys" Jul 10 17:18:33.260754 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.260716 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.263642 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.263622 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-rmfx2\"" Jul 10 17:18:33.263768 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.263673 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-fd7cfb-serving-cert\"" Jul 10 17:18:33.263923 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.263910 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 10 17:18:33.264849 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.264832 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 10 17:18:33.273722 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.273697 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:18:33.393712 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.393672 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.393899 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.393730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.393899 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.393767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cbk\" (UniqueName: \"kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.494294 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.494260 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.494482 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.494318 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.494482 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.494338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cbk\" (UniqueName: \"kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.495010 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.494985 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.496689 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.496672 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.502311 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.502286 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cbk\" (UniqueName: \"kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk\") pod \"maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.571104 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.571001 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:33.712072 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:33.712047 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:18:33.714426 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:18:33.714399 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62414ca6_456e_4b9d_a9dc_cc4fc84486bf.slice/crio-7401d1b0bace380b4f42e52f03315808a8cb05c36fd193ec618c1ecf84e3922b WatchSource:0}: Error finding container 7401d1b0bace380b4f42e52f03315808a8cb05c36fd193ec618c1ecf84e3922b: Status 404 returned error can't find the container with id 7401d1b0bace380b4f42e52f03315808a8cb05c36fd193ec618c1ecf84e3922b Jul 10 17:18:34.713160 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:34.713123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" event={"ID":"62414ca6-456e-4b9d-a9dc-cc4fc84486bf","Type":"ContainerStarted","Data":"c79118a17eba7eda7fdebd8147831d8606ed91fbc560a1eda26c9fde6b7cb1fd"} Jul 10 17:18:34.713548 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:34.713176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" event={"ID":"62414ca6-456e-4b9d-a9dc-cc4fc84486bf","Type":"ContainerStarted","Data":"7401d1b0bace380b4f42e52f03315808a8cb05c36fd193ec618c1ecf84e3922b"} Jul 10 17:18:34.713548 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:34.713209 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:34.733687 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:34.733631 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" podStartSLOduration=1.345780148 podStartE2EDuration="1.733613561s" totalImagesPullingTime="387.833413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:34.73155419 +0000 UTC m=+2504.407615974" watchObservedRunningTime="2026-07-10 17:18:34.733613561 +0000 UTC m=+2504.409675344" Jul 10 17:18:40.723407 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:40.723378 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:18:41.494507 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.494409 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:18:41.498609 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.498591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.501126 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.501101 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c9789a-serving-cert\"" Jul 10 17:18:41.510249 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.510222 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:18:41.665249 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.665213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4h6\" (UniqueName: \"kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.665410 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.665280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.665410 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.665336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.766063 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.765980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.766063 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.766044 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.766483 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.766076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4h6\" (UniqueName: \"kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.766483 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:18:41.766129 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-serving-cert: secret "maas-api-e2e-shared-b-c9789a-serving-cert" not found Jul 10 17:18:41.766483 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:18:41.766196 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls podName:7aef985b-d4a2-45db-a9f5-9c97eb3b9356 nodeName:}" failed. No retries permitted until 2026-07-10 17:18:42.266175632 +0000 UTC m=+2511.942237408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls") pod "maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" (UID: "7aef985b-d4a2-45db-a9f5-9c97eb3b9356") : secret "maas-api-e2e-shared-b-c9789a-serving-cert" not found Jul 10 17:18:41.766701 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.766678 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:41.776662 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:41.776626 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4h6\" (UniqueName: \"kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:42.270436 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:42.270392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:42.272851 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:42.272831 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") pod \"maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:42.412420 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:42.412381 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:42.557172 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:42.557145 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:18:42.559120 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:18:42.559092 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aef985b_d4a2_45db_a9f5_9c97eb3b9356.slice/crio-a06f99dc21c02da9ade76ad486e4bb0e6524dc5c933d7a3a9c7db63d89221d74 WatchSource:0}: Error finding container a06f99dc21c02da9ade76ad486e4bb0e6524dc5c933d7a3a9c7db63d89221d74: Status 404 returned error can't find the container with id a06f99dc21c02da9ade76ad486e4bb0e6524dc5c933d7a3a9c7db63d89221d74 Jul 10 17:18:42.757859 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:42.757823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" event={"ID":"7aef985b-d4a2-45db-a9f5-9c97eb3b9356","Type":"ContainerStarted","Data":"a06f99dc21c02da9ade76ad486e4bb0e6524dc5c933d7a3a9c7db63d89221d74"} Jul 10 17:18:43.763720 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:43.763688 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" event={"ID":"7aef985b-d4a2-45db-a9f5-9c97eb3b9356","Type":"ContainerStarted","Data":"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d"} Jul 10 17:18:43.764142 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:43.763830 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:18:43.784356 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:43.784302 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" podStartSLOduration=2.417444452 podStartE2EDuration="2.784285269s" totalImagesPullingTime="366.840817ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-10 17:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:18:43.782399745 +0000 UTC m=+2513.458461529" watchObservedRunningTime="2026-07-10 17:18:43.784285269 +0000 UTC m=+2513.460347051" Jul 10 17:18:49.774662 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:18:49.774632 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:21:51.015100 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:21:51.015072 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:21:51.021557 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:21:51.021536 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-scsl6_9bdf848b-cc24-4998-bb53-4a3ea919e2f6/ovn-acl-logging/0.log" Jul 10 17:25:18.294941 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.294895 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:25:18.295783 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.295707 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" podUID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" containerName="maas-api" containerID="cri-o://c79118a17eba7eda7fdebd8147831d8606ed91fbc560a1eda26c9fde6b7cb1fd" gracePeriod=30 Jul 10 17:25:18.649791 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.649734 2568 generic.go:356] "Generic (PLEG): container finished" podID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" containerID="c79118a17eba7eda7fdebd8147831d8606ed91fbc560a1eda26c9fde6b7cb1fd" exitCode=0 Jul 10 17:25:18.649989 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.649808 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" event={"ID":"62414ca6-456e-4b9d-a9dc-cc4fc84486bf","Type":"ContainerDied","Data":"c79118a17eba7eda7fdebd8147831d8606ed91fbc560a1eda26c9fde6b7cb1fd"} Jul 10 17:25:18.755278 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.755246 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:25:18.930160 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.930122 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7cbk\" (UniqueName: \"kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk\") pod \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " Jul 10 17:25:18.930351 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.930195 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca\") pod \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " Jul 10 17:25:18.930351 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.930234 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls\") pod \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\" (UID: \"62414ca6-456e-4b9d-a9dc-cc4fc84486bf\") " Jul 10 17:25:18.930506 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.930482 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62414ca6-456e-4b9d-a9dc-cc4fc84486bf" (UID: "62414ca6-456e-4b9d-a9dc-cc4fc84486bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:25:18.932322 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.932292 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62414ca6-456e-4b9d-a9dc-cc4fc84486bf" (UID: "62414ca6-456e-4b9d-a9dc-cc4fc84486bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:25:18.932620 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:18.932599 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk" (OuterVolumeSpecName: "kube-api-access-r7cbk") pod "62414ca6-456e-4b9d-a9dc-cc4fc84486bf" (UID: "62414ca6-456e-4b9d-a9dc-cc4fc84486bf"). InnerVolumeSpecName "kube-api-access-r7cbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:19.031856 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.031817 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:19.031856 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.031850 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7cbk\" (UniqueName: \"kubernetes.io/projected/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-kube-api-access-r7cbk\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:19.031856 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.031861 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62414ca6-456e-4b9d-a9dc-cc4fc84486bf-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:19.656658 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.656622 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" event={"ID":"62414ca6-456e-4b9d-a9dc-cc4fc84486bf","Type":"ContainerDied","Data":"7401d1b0bace380b4f42e52f03315808a8cb05c36fd193ec618c1ecf84e3922b"} Jul 10 17:25:19.657095 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.656681 2568 scope.go:117] "RemoveContainer" containerID="c79118a17eba7eda7fdebd8147831d8606ed91fbc560a1eda26c9fde6b7cb1fd" Jul 10 17:25:19.657095 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.656631 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p" Jul 10 17:25:19.676313 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.676290 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:25:19.680518 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:19.680493 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-fd7cfb-5d47cb497b-v9x2p"] Jul 10 17:25:21.044470 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:21.044437 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" path="/var/lib/kubelet/pods/62414ca6-456e-4b9d-a9dc-cc4fc84486bf/volumes" Jul 10 17:25:34.030365 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.030331 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx"] Jul 10 17:25:34.032571 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.030659 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" containerName="maas-api" Jul 10 17:25:34.032571 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.030669 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" containerName="maas-api" Jul 10 17:25:34.032571 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.030728 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="62414ca6-456e-4b9d-a9dc-cc4fc84486bf" containerName="maas-api" Jul 10 17:25:34.033526 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.033511 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:34.036528 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.036500 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-m9lhw\"" Jul 10 17:25:34.043667 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.043643 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx"] Jul 10 17:25:34.047316 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.047287 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx\" (UID: \"f51dc416-b3c3-40d1-a5a5-74ba03aba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:34.148718 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.148678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx\" (UID: \"f51dc416-b3c3-40d1-a5a5-74ba03aba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:34.149322 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.149303 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx\" (UID: \"f51dc416-b3c3-40d1-a5a5-74ba03aba061\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:34.345637 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.345562 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:34.477419 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.477386 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx"] Jul 10 17:25:34.479978 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:25:34.479929 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51dc416_b3c3_40d1_a5a5_74ba03aba061.slice/crio-6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548 WatchSource:0}: Error finding container 6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548: Status 404 returned error can't find the container with id 6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548 Jul 10 17:25:34.728156 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:34.728126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" event={"ID":"f51dc416-b3c3-40d1-a5a5-74ba03aba061","Type":"ContainerStarted","Data":"6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548"} Jul 10 17:25:35.736113 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:35.736080 2568 generic.go:356] "Generic (PLEG): container finished" podID="f51dc416-b3c3-40d1-a5a5-74ba03aba061" containerID="e507379e74b54b8baedd2d8725194def9d5c310220da9acc74e169e9cd130280" exitCode=0 Jul 10 17:25:35.736496 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:35.736124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" event={"ID":"f51dc416-b3c3-40d1-a5a5-74ba03aba061","Type":"ContainerDied","Data":"e507379e74b54b8baedd2d8725194def9d5c310220da9acc74e169e9cd130280"} Jul 10 17:25:36.870635 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:36.870610 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:36.970354 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:36.970324 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca\") pod \"f51dc416-b3c3-40d1-a5a5-74ba03aba061\" (UID: \"f51dc416-b3c3-40d1-a5a5-74ba03aba061\") " Jul 10 17:25:36.970677 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:36.970656 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f51dc416-b3c3-40d1-a5a5-74ba03aba061" (UID: "f51dc416-b3c3-40d1-a5a5-74ba03aba061"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:25:37.071469 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:37.071433 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f51dc416-b3c3-40d1-a5a5-74ba03aba061-maas-api-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:37.747162 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:37.747127 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" event={"ID":"f51dc416-b3c3-40d1-a5a5-74ba03aba061","Type":"ContainerDied","Data":"6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548"} Jul 10 17:25:37.747162 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:37.747152 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx" Jul 10 17:25:37.747365 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:37.747180 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4d4d1f08fa4c98e30bcdae37f8b27ea2250e1c7542ed097e70d740007d6548" Jul 10 17:25:44.431121 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.431085 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:25:44.431568 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.431333 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" podUID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" containerName="maas-api" containerID="cri-o://8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d" gracePeriod=30 Jul 10 17:25:44.692969 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.692903 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:25:44.729830 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.729800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") pod \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " Jul 10 17:25:44.729984 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.729849 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca\") pod \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " Jul 10 17:25:44.729984 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.729890 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf4h6\" (UniqueName: \"kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6\") pod \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\" (UID: \"7aef985b-d4a2-45db-a9f5-9c97eb3b9356\") " Jul 10 17:25:44.730229 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.730204 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7aef985b-d4a2-45db-a9f5-9c97eb3b9356" (UID: "7aef985b-d4a2-45db-a9f5-9c97eb3b9356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 10 17:25:44.732035 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.731994 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6" (OuterVolumeSpecName: "kube-api-access-gf4h6") pod "7aef985b-d4a2-45db-a9f5-9c97eb3b9356" (UID: "7aef985b-d4a2-45db-a9f5-9c97eb3b9356"). InnerVolumeSpecName "kube-api-access-gf4h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 10 17:25:44.734447 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.734410 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7aef985b-d4a2-45db-a9f5-9c97eb3b9356" (UID: "7aef985b-d4a2-45db-a9f5-9c97eb3b9356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 10 17:25:44.784121 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.784083 2568 generic.go:356] "Generic (PLEG): container finished" podID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" containerID="8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d" exitCode=0 Jul 10 17:25:44.784250 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.784148 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" Jul 10 17:25:44.784250 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.784175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" event={"ID":"7aef985b-d4a2-45db-a9f5-9c97eb3b9356","Type":"ContainerDied","Data":"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d"} Jul 10 17:25:44.784368 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.784270 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6" event={"ID":"7aef985b-d4a2-45db-a9f5-9c97eb3b9356","Type":"ContainerDied","Data":"a06f99dc21c02da9ade76ad486e4bb0e6524dc5c933d7a3a9c7db63d89221d74"} Jul 10 17:25:44.784368 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.784303 2568 scope.go:117] "RemoveContainer" containerID="8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d" Jul 10 17:25:44.793647 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.793633 2568 scope.go:117] "RemoveContainer" containerID="8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d" Jul 10 17:25:44.793934 ip-10-0-131-107 kubenswrapper[2568]: E0710 17:25:44.793916 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d\": container with ID starting with 8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d not found: ID does not exist" containerID="8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d" Jul 10 17:25:44.794004 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.793941 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d"} err="failed to get container status \"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d\": rpc error: code = NotFound desc = could not find container \"8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d\": container with ID starting with 8c723ec8b981e6ffa4c01ed1142c7cc3eca2c6b48cd91aa76be5d5fbfe31c56d not found: ID does not exist" Jul 10 17:25:44.807118 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.807091 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:25:44.809226 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.809208 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c9789a-77cb65bd74-xdpb6"] Jul 10 17:25:44.830633 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.830614 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-openshift-service-ca\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:44.830633 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.830633 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4h6\" (UniqueName: \"kubernetes.io/projected/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-kube-api-access-gf4h6\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:44.830767 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:44.830644 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7aef985b-d4a2-45db-a9f5-9c97eb3b9356-maas-api-tls\") on node \"ip-10-0-131-107.ec2.internal\" DevicePath \"\"" Jul 10 17:25:45.043690 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:25:45.043611 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" path="/var/lib/kubelet/pods/7aef985b-d4a2-45db-a9f5-9c97eb3b9356/volumes" Jul 10 17:26:05.431957 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:05.431924 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5h2lb_9cd1b184-2cb5-4d92-a025-94923e4de14a/manager/0.log" Jul 10 17:26:05.561001 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:05.560954 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bq6n5_adc7a28b-4499-42dc-af21-f50804262a30/manager/0.log" Jul 10 17:26:05.817882 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:05.817796 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9r6j6_7832425b-88e9-4b4b-9447-66846bc99088/server/0.log" Jul 10 17:26:05.957465 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:05.957439 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8gpz5_da915a25-efba-4b15-b7af-492200ad2c50/manager/0.log" Jul 10 17:26:06.204965 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:06.204943 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-qmq6c_1adbb660-efa8-4bd2-aef0-ba129d2a011d/copy-manifests/0.log" Jul 10 17:26:06.213430 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:06.213404 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-qmq6c_1adbb660-efa8-4bd2-aef0-ba129d2a011d/manager/0.log" Jul 10 17:26:06.841035 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:06.841014 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728380-z8wpk_d6fc98fb-4ce1-437e-958f-dd8609341a28/cleanup/0.log" Jul 10 17:26:06.963553 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:06.963523 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29728395-pckp7_4ab27482-63c8-478f-a934-b2540649241c/cleanup/0.log" Jul 10 17:26:07.071550 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.071522 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-7c0fd7dc-pkvf6_61d1f551-093b-482a-b151-d9e377cff94f/revoke-keys/0.log" Jul 10 17:26:07.182306 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.182284 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-fdbb8df2-cdsnk_4fc4c0b1-31d6-45b5-8cbe-5c2fbb27cf2a/revoke-keys/0.log" Jul 10 17:26:07.422012 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.421981 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-dd88b4-c7j7w_6c4e8f8c-b9f0-494f-a914-5d241616590c/revoke-keys/0.log" Jul 10 17:26:07.534480 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.534391 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-ef0ffa22-hgmzj_df36ef77-be30-4c1a-a515-b4c1fd35debf/revoke-keys/0.log" Jul 10 17:26:07.645621 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.645593 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-889e72-hx7cz_beb1c8b2-722c-45c9-911e-2e3250c3f7a5/revoke-keys/0.log" Jul 10 17:26:07.755768 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.755728 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-a0f0a2-4xpkf_c27da0ed-b10f-4887-9abd-29b15ee8f5d9/revoke-keys/0.log" Jul 10 17:26:07.861450 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.861368 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-af50a058-m5qzq_92c98935-6ce2-4c6a-bd7b-066a2ce4471c/revoke-keys/0.log" Jul 10 17:26:07.965077 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:07.965051 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-e08efa8c-xtmvm_dd2dc421-a448-474a-a2ec-1c82287bbbd6/revoke-keys/0.log" Jul 10 17:26:08.196505 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:08.196479 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-c9789a-jl9lx_f51dc416-b3c3-40d1-a5a5-74ba03aba061/revoke-keys/0.log" Jul 10 17:26:09.814493 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:09.814463 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8m85g_0f810fb3-ebfc-4a93-a070-848831617a8c/manager/0.log" Jul 10 17:26:10.274719 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:10.274690 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-b6h7s_8b223535-8f86-485b-8052-28dcfd85f04c/manager/0.log" Jul 10 17:26:11.061645 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:11.061613 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-sk8nx_e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8/discovery/0.log" Jul 10 17:26:11.175923 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:11.175893 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5778d5b958-dwwbz_8deaa765-28bb-4d0c-a71f-759bf8b6b639/kube-auth-proxy/0.log" Jul 10 17:26:12.235570 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:12.235540 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-b6sq8_cfb04eb5-a83f-4526-8950-29246c34b620/main/0.log" Jul 10 17:26:12.569416 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:12.569328 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-cpffh_11be1cc1-6075-480a-a975-6b0055f6e89f/main/0.log" Jul 10 17:26:19.867552 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:19.867519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nhcft_29764102-4050-48c5-bc52-cc0e03dfc516/global-pull-secret-syncer/0.log" Jul 10 17:26:20.027277 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:20.027247 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xmdzd_0e0ea746-b225-44d5-9981-db1ecfc29ca5/konnectivity-agent/0.log" Jul 10 17:26:20.052116 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:20.052090 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-107.ec2.internal_44bd8c551779de4c442e1d9ae5fc2be1/haproxy/0.log" Jul 10 17:26:24.480109 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:24.480076 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8m85g_0f810fb3-ebfc-4a93-a070-848831617a8c/manager/0.log" Jul 10 17:26:24.788713 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:24.788612 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-b6h7s_8b223535-8f86-485b-8052-28dcfd85f04c/manager/0.log" Jul 10 17:26:27.042953 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:27.042907 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcs5s_a62d1948-0dfd-4404-ae42-daf90b9d5341/node-exporter/0.log" Jul 10 17:26:27.072836 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:27.072807 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcs5s_a62d1948-0dfd-4404-ae42-daf90b9d5341/kube-rbac-proxy/0.log" Jul 10 17:26:27.096389 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:27.096363 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fcs5s_a62d1948-0dfd-4404-ae42-daf90b9d5341/init-textfile/0.log" Jul 10 17:26:30.204557 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.204520 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j"] Jul 10 17:26:30.205062 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205024 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f51dc416-b3c3-40d1-a5a5-74ba03aba061" containerName="revoke-keys" Jul 10 17:26:30.205062 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205041 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f51dc416-b3c3-40d1-a5a5-74ba03aba061" containerName="revoke-keys" Jul 10 17:26:30.205183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205075 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" containerName="maas-api" Jul 10 17:26:30.205183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205085 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" containerName="maas-api" Jul 10 17:26:30.205183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205169 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aef985b-d4a2-45db-a9f5-9c97eb3b9356" containerName="maas-api" Jul 10 17:26:30.205183 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.205182 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f51dc416-b3c3-40d1-a5a5-74ba03aba061" containerName="revoke-keys" Jul 10 17:26:30.208472 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.208434 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.210795 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.210774 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq8tx\"/\"kube-root-ca.crt\"" Jul 10 17:26:30.211883 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.211859 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bq8tx\"/\"default-dockercfg-w74x2\"" Jul 10 17:26:30.211996 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.211859 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bq8tx\"/\"openshift-service-ca.crt\"" Jul 10 17:26:30.217972 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.217340 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-gxtzv_765508fe-517c-4c4a-a954-734c2573ede7/download-server/0.log" Jul 10 17:26:30.220083 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.220061 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j"] Jul 10 17:26:30.305942 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.305909 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-proc\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.306119 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.305958 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-lib-modules\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.306119 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.305982 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lfff\" (UniqueName: \"kubernetes.io/projected/52bb009e-f8e9-498d-be91-a4a49e561ecd-kube-api-access-7lfff\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.306119 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.306058 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-sys\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.306119 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.306091 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-podres\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406478 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-sys\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406478 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406481 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-podres\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-proc\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406565 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-lib-modules\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-sys\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lfff\" (UniqueName: \"kubernetes.io/projected/52bb009e-f8e9-498d-be91-a4a49e561ecd-kube-api-access-7lfff\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406668 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-proc\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.406706 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406684 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-lib-modules\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.407023 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.406814 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52bb009e-f8e9-498d-be91-a4a49e561ecd-podres\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.416161 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.416141 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lfff\" (UniqueName: \"kubernetes.io/projected/52bb009e-f8e9-498d-be91-a4a49e561ecd-kube-api-access-7lfff\") pod \"perf-node-gather-daemonset-p2l5j\" (UID: \"52bb009e-f8e9-498d-be91-a4a49e561ecd\") " pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.522018 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.521915 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:30.652311 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:30.652286 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j"] Jul 10 17:26:30.653802 ip-10-0-131-107 kubenswrapper[2568]: W0710 17:26:30.653777 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52bb009e_f8e9_498d_be91_a4a49e561ecd.slice/crio-65b3dd5ffd460ad6e39c316df3e7e19958b2ff7e7ed35040fa169c5fda198066 WatchSource:0}: Error finding container 65b3dd5ffd460ad6e39c316df3e7e19958b2ff7e7ed35040fa169c5fda198066: Status 404 returned error can't find the container with id 65b3dd5ffd460ad6e39c316df3e7e19958b2ff7e7ed35040fa169c5fda198066 Jul 10 17:26:31.011244 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.011206 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" event={"ID":"52bb009e-f8e9-498d-be91-a4a49e561ecd","Type":"ContainerStarted","Data":"b16deca20ab036b2fd42504ac8c922f97b8346e34c755657a98944c0a2ea7586"} Jul 10 17:26:31.011442 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.011264 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" event={"ID":"52bb009e-f8e9-498d-be91-a4a49e561ecd","Type":"ContainerStarted","Data":"65b3dd5ffd460ad6e39c316df3e7e19958b2ff7e7ed35040fa169c5fda198066"} Jul 10 17:26:31.011442 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.011285 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:31.028560 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.028511 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" podStartSLOduration=1.028493875 podStartE2EDuration="1.028493875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-10 17:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-10 17:26:31.025903118 +0000 UTC m=+2980.701964902" watchObservedRunningTime="2026-07-10 17:26:31.028493875 +0000 UTC m=+2980.704555659" Jul 10 17:26:31.910031 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.909998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8cdld_b397527c-1fa8-415d-ac0e-61dadab748f1/dns/0.log" Jul 10 17:26:31.930725 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:31.930696 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8cdld_b397527c-1fa8-415d-ac0e-61dadab748f1/kube-rbac-proxy/0.log" Jul 10 17:26:32.059790 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:32.059759 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-2ndkl_ff236cfe-018a-42db-a189-3b1d84b276cf/dns-node-resolver/0.log" Jul 10 17:26:32.703721 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:32.703693 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-692mv_a3758299-2838-43fd-ad6a-bc8b885579cd/node-ca/0.log" Jul 10 17:26:33.896721 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:33.896688 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-sk8nx_e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8/discovery/0.log" Jul 10 17:26:33.918969 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:33.918934 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5778d5b958-dwwbz_8deaa765-28bb-4d0c-a71f-759bf8b6b639/kube-auth-proxy/0.log" Jul 10 17:26:34.778934 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:34.778900 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2wq72_a74daaba-34e5-48e5-92b0-70066cce4536/serve-healthcheck-canary/0.log" Jul 10 17:26:35.526428 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:35.526403 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4qbg_dc49631f-2768-44ba-815b-3603f99bd9ea/kube-rbac-proxy/0.log" Jul 10 17:26:35.551179 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:35.551154 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4qbg_dc49631f-2768-44ba-815b-3603f99bd9ea/exporter/0.log" Jul 10 17:26:35.593947 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:35.593918 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m4qbg_dc49631f-2768-44ba-815b-3603f99bd9ea/extractor/0.log" Jul 10 17:26:37.031722 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:37.031688 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bq8tx/perf-node-gather-daemonset-p2l5j" Jul 10 17:26:37.752091 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:37.752062 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5h2lb_9cd1b184-2cb5-4d92-a025-94923e4de14a/manager/0.log" Jul 10 17:26:37.988327 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:37.988292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bq6n5_adc7a28b-4499-42dc-af21-f50804262a30/manager/0.log" Jul 10 17:26:38.109271 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:38.109190 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-9r6j6_7832425b-88e9-4b4b-9447-66846bc99088/server/0.log" Jul 10 17:26:38.211358 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:38.211327 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8gpz5_da915a25-efba-4b15-b7af-492200ad2c50/manager/0.log" Jul 10 17:26:38.287412 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:38.287376 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-qmq6c_1adbb660-efa8-4bd2-aef0-ba129d2a011d/manager/0.log" Jul 10 17:26:38.307594 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:38.307571 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-585db49996-qmq6c_1adbb660-efa8-4bd2-aef0-ba129d2a011d/copy-manifests/0.log" Jul 10 17:26:39.877683 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:39.877605 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-sfrvm_388a8ca1-b816-4916-a106-a2c895480497/openshift-lws-operator/0.log" Jul 10 17:26:40.116783 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:40.116756 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-sk8nx_e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8/discovery/0.log" Jul 10 17:26:40.128223 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:40.128141 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5778d5b958-dwwbz_8deaa765-28bb-4d0c-a71f-759bf8b6b639/kube-auth-proxy/0.log" Jul 10 17:26:41.802251 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:41.802212 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-sk8nx_e3f3223c-a28b-4e9c-9a7a-5b4efd9333c8/discovery/0.log" Jul 10 17:26:41.817010 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:41.816985 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5778d5b958-dwwbz_8deaa765-28bb-4d0c-a71f-759bf8b6b639/kube-auth-proxy/0.log" Jul 10 17:26:46.486403 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.486373 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/kube-multus-additional-cni-plugins/0.log" Jul 10 17:26:46.508682 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.508651 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/egress-router-binary-copy/0.log" Jul 10 17:26:46.530912 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.530886 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/cni-plugins/0.log" Jul 10 17:26:46.552483 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.552458 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/bond-cni-plugin/0.log" Jul 10 17:26:46.573263 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.573237 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/routeoverride-cni/0.log" Jul 10 17:26:46.595223 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.595195 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/whereabouts-cni-bincopy/0.log" Jul 10 17:26:46.616484 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.616461 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-r276w_ec6e9897-b9bd-4f00-acbd-72aea3e101e7/whereabouts-cni/0.log" Jul 10 17:26:46.897393 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.897311 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-zr4fb_70523d1b-ee7e-495d-a301-ade04b748746/kube-multus/0.log" Jul 10 17:26:46.969798 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.969772 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8mssf_cadf1967-b262-4bdd-a933-968447f88cf1/network-metrics-daemon/0.log" Jul 10 17:26:46.990786 ip-10-0-131-107 kubenswrapper[2568]: I0710 17:26:46.990732 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8mssf_cadf1967-b262-4bdd-a933-968447f88cf1/kube-rbac-proxy/0.log"