Jul 08 13:20:55.939163 ip-10-0-138-212 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 08 13:20:55.939176 ip-10-0-138-212 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 08 13:20:55.939185 ip-10-0-138-212 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 08 13:20:55.939609 ip-10-0-138-212 systemd[1]: Failed to start Kubernetes Kubelet. Jul 08 13:21:05.976685 ip-10-0-138-212 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 08 13:21:05.976700 ip-10-0-138-212 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot af28ad270791401bbaac69f50b9047ef -- Jul 08 13:22:40.133344 ip-10-0-138-212 systemd[1]: Starting Kubernetes Kubelet... Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 13:22:40.590619 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.588854 2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594346 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594358 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594361 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594366 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594370 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594372 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594375 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594378 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594381 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594385 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594387 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594390 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594392 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594395 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594398 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594401 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594403 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594406 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594409 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:40.627587 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594412 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594414 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594419 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594423 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594426 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594429 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594432 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594435 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594437 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594440 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594443 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594446 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594448 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594451 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594453 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594455 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594458 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594461 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594464 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594466 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:40.646801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594469 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594471 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594473 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594476 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594478 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594481 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594483 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594486 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594488 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594490 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594495 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594498 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594501 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594504 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594506 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594509 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594512 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594525 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594531 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:40.896867 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594534 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:40.667464 ip-10-0-138-212 systemd[1]: Started Kubernetes Kubelet. Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594537 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594541 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594545 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594548 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594552 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594555 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594558 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594560 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594563 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594566 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594569 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594572 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594575 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594577 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594580 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594582 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594585 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594587 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:40.907938 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594590 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594593 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594595 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594598 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594600 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594603 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594605 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594608 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594610 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594613 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594615 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594618 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594621 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594624 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594626 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594629 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594632 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594634 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594637 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594640 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:40.908699 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594642 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.594645 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597881 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597892 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597896 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597900 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597905 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597909 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597914 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597916 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597919 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597922 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597925 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597928 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597931 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597934 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597936 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597940 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597944 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597948 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:40.909615 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597950 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597954 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597956 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597959 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597961 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597964 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597966 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597969 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597972 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597975 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597977 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597980 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597982 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597986 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597989 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597991 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597994 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597997 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.597999 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598002 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:40.910362 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598005 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598007 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598010 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598012 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598015 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598018 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598020 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598023 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598026 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598029 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598031 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598034 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598037 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598040 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598043 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598045 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598047 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598050 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598074 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598077 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:40.911942 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598080 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598082 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598085 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598088 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598091 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598093 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598096 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598100 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598104 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598107 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598109 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598112 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598114 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598117 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598119 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598122 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598124 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598128 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598131 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:40.913737 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598133 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598136 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598139 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598141 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598145 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598149 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598153 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598156 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598159 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598161 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598165 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598167 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598170 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598172 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598175 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598177 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598180 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598182 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598185 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:40.914709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598187 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598190 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598192 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598563 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598569 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598574 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598577 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598580 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598583 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598586 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598589 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598591 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598594 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598597 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598601 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598604 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598606 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598609 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598611 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:40.916711 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598616 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598619 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598622 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598624 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598627 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598630 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598632 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598635 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598638 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598641 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598643 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598646 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598649 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598651 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598654 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598657 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598659 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598662 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598664 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598667 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:40.919041 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598669 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598672 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598674 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598677 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598681 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598684 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598687 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598690 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598692 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598695 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598697 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598700 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598704 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598707 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598709 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598712 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598716 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598719 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598721 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:40.921310 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598724 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598726 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598729 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598731 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598734 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598736 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598739 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598741 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598744 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598747 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598749 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598752 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598755 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598757 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598759 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598762 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598765 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598768 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598770 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598773 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:40.922395 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598775 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598778 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598781 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598783 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598786 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598789 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598791 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598795 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598798 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598802 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598806 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598809 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598812 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598815 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598818 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598821 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598823 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598826 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598828 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598831 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:40.924564 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598834 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598837 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598839 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.598842 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598909 2576 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598919 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598926 2576 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598930 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598934 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598937 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598942 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598946 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598949 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598952 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598955 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598958 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598962 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598966 2576 flags.go:64] FLAG: --cgroup-root="" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598969 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598972 2576 flags.go:64] FLAG: --client-ca-file="" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598975 2576 flags.go:64] FLAG: --cloud-provider="external" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598980 2576 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598984 2576 flags.go:64] FLAG: --cluster-domain="" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598987 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 13:22:40.925767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598990 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598993 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.598996 2576 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599004 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599007 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599011 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599014 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599017 2576 flags.go:64] FLAG: --contention-profiling="false" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599020 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599023 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599026 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599029 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599033 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599036 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599039 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599042 2576 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599044 2576 flags.go:64] FLAG: --enable-server="true" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599047 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599068 2576 flags.go:64] FLAG: --event-burst="100" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599072 2576 flags.go:64] FLAG: --event-qps="50" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599075 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599078 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599081 2576 flags.go:64] FLAG: --eviction-hard="" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599085 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 13:22:40.927009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599088 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599093 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599096 2576 flags.go:64] FLAG: --eviction-soft="" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599099 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599101 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599106 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599109 2576 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599112 2576 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599115 2576 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599118 2576 flags.go:64] FLAG: --feature-gates="" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599122 2576 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599125 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599128 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599131 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599134 2576 flags.go:64] FLAG: --healthz-port="10248" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599136 2576 flags.go:64] FLAG: --help="false" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599139 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-138-212.ec2.internal" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599142 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599145 2576 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599147 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599150 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599154 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599157 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 13:22:40.928027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599160 2576 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599162 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599166 2576 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599168 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599171 2576 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599174 2576 flags.go:64] FLAG: --kube-reserved="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599177 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599179 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599182 2576 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599185 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599190 2576 flags.go:64] FLAG: --lock-file="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599192 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599195 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599198 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599205 2576 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599209 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599211 2576 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599214 2576 flags.go:64] FLAG: --logging-format="text" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599217 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599220 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599223 2576 flags.go:64] FLAG: --manifest-url="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599226 2576 flags.go:64] FLAG: --manifest-url-header="" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599230 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599234 2576 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599238 2576 flags.go:64] FLAG: --max-pods="110" Jul 08 13:22:40.929379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599241 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599244 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599246 2576 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599249 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599252 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599255 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599258 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599264 2576 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599267 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599270 2576 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599273 2576 flags.go:64] FLAG: --pod-cidr="" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599276 2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599282 2576 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599285 2576 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599288 2576 flags.go:64] FLAG: --pods-per-core="0" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599291 2576 flags.go:64] FLAG: --port="10250" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599293 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599298 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09ca8c1d33cb15475" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599301 2576 flags.go:64] FLAG: --qos-reserved="" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599304 2576 flags.go:64] FLAG: --read-only-port="10255" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599307 2576 flags.go:64] FLAG: --register-node="true" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599310 2576 flags.go:64] FLAG: --register-with-taints="" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599315 2576 flags.go:64] FLAG: --registry-burst="10" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599318 2576 flags.go:64] FLAG: --registry-qps="5" Jul 08 13:22:40.930534 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599321 2576 flags.go:64] FLAG: --reserved-cpus="" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599324 2576 flags.go:64] FLAG: --reserved-memory="" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599328 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599331 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599334 2576 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599337 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599339 2576 flags.go:64] FLAG: --runonce="false" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599343 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599345 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599348 2576 flags.go:64] FLAG: --seccomp-default="false" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599352 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599354 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599357 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599360 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599363 2576 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599366 2576 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599369 2576 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599372 2576 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599374 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599377 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599380 2576 flags.go:64] FLAG: --system-cgroups="" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599383 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599388 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599390 2576 flags.go:64] FLAG: --tls-cert-file="" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599393 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 13:22:40.931665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599399 2576 flags.go:64] FLAG: --tls-min-version="" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599402 2576 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599405 2576 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599408 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599411 2576 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599414 2576 flags.go:64] FLAG: --v="2" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599418 2576 flags.go:64] FLAG: --version="false" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599423 2576 flags.go:64] FLAG: --vmodule="" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599426 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.599430 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599541 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599545 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599548 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599551 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599554 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599556 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599559 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599562 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599565 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599568 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599570 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599573 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:40.932758 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599575 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599578 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599581 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599583 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599586 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599588 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599591 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599594 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599596 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599599 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599602 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599605 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599607 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599610 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599613 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599616 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599619 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599621 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599624 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599627 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:40.935543 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599629 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599632 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599634 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599637 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599639 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599642 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599644 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599646 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599649 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599651 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599654 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599657 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599659 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599662 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599664 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599667 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599669 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599672 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599675 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599677 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:40.936410 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599680 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599684 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599689 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599692 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599694 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599698 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599702 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599707 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599710 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599713 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599716 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599719 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599722 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599725 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599727 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599730 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599732 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599735 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599737 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599740 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:40.937718 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599742 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599745 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599747 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599750 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599752 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599756 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599759 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599762 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599765 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599767 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599770 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599772 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599775 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599778 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599782 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599784 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599787 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599789 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599791 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:40.938604 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599795 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599798 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599800 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599802 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599805 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599808 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599811 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.599813 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.600784 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.609287 2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.609305 2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.609328 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.609335 2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609398 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609404 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:40.939680 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609408 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609412 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609415 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609420 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609426 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609429 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609432 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609437 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609440 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609443 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609446 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609448 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609451 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609455 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609458 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609462 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609465 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609467 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609470 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:40.940353 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609475 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609477 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609480 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609483 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609487 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609490 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609494 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609497 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609501 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609504 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609507 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609512 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609515 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609520 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609523 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609526 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609530 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609533 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609536 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609539 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:40.944639 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609543 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609545 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609548 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609550 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609554 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609557 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609560 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609564 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609567 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609570 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609573 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609577 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609580 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609582 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609585 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609589 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609592 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609594 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609598 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609602 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:40.945525 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609605 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609613 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609616 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609619 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609623 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609626 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609629 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609633 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609639 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609643 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609647 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609650 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609653 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609656 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609659 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609661 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609664 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609668 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609670 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:40.946846 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609673 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609676 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609680 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609683 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609685 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609689 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609692 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609695 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609697 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609701 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609704 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609706 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609709 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609715 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609718 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609721 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609725 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609728 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:40.947710 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609731 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.609737 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609862 2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609868 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609872 2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609875 2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609878 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609881 2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609885 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609888 2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609890 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609894 2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609897 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609900 2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609903 2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 13:22:41.042029 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609906 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609909 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609912 2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609915 2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609918 2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609921 2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609924 2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609927 2576 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609930 2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609933 2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609935 2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609938 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609942 2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609946 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609949 2576 feature_gate.go:328] unrecognized feature gate: Example Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609953 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609956 2576 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609958 2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609962 2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609966 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 13:22:41.046662 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609970 2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609975 2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609979 2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609982 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609985 2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609989 2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609991 2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.609996 2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610000 2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610004 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610007 2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610011 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610014 2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610018 2576 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610021 2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610025 2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610027 2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610031 2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610034 2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 13:22:41.047739 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610036 2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610039 2576 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610042 2576 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610045 2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610048 2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610066 2576 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610070 2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610073 2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610076 2576 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610080 2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610083 2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610086 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610089 2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610094 2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610097 2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610100 2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610104 2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610106 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610109 2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610112 2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 13:22:41.048601 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610115 2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610118 2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610120 2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610123 2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610126 2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610129 2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610132 2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610135 2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610139 2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610142 2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610145 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610149 2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610152 2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610155 2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610157 2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610161 2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610164 2576 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610167 2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610171 2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 13:22:41.050313 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610175 2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610179 2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610182 2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610187 2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610190 2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610193 2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610201 2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:40.610204 2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.610210 2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.610327 2576 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.613459 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.615096 2576 server.go:1019] "Starting client certificate rotation" Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.615200 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 13:22:41.051393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.615230 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.617822 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.619954 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.633510 2576 log.go:25] "Validated CRI v1 runtime API" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.635824 2576 log.go:25] "Validated CRI v1 image API" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.636606 2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.639874 2576 fs.go:135] Filesystem UUIDs: map[60186a82-42e3-48d3-b189-0e93550533ae:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b9fd36c6-730d-47d2-a73d-1d596bf986af:/dev/nvme0n1p3] Jul 08 13:22:41.052035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.639887 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.644438 2576 manager.go:216] Machine: {Timestamp:2026-07-08 13:22:40.642965803 +0000 UTC m=+0.374803683 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3143914 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2d470ec141627615c93cd1e1c17c8e SystemUUID:ec2d470e-c141-6276-15c9-3cd1e1c17c8e BootID:af28ad27-0791-401b-baac-69f50b9047ef Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f5:c7:82:8a:25 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f5:c7:82:8a:25 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:ec:06:21:43:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.644544 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.644620 2576 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.647959 2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.647983 2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-212.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.648875 2576 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.648884 2576 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.648891 2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.648902 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.649779 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.650863 2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654364 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654467 2576 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654493 2576 kubelet.go:486] "Attempting to sync node with API server" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654501 2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654526 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 13:22:41.053951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654541 2576 kubelet.go:398] "Adding apiserver pod source" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654604 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654902 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.654916 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.657382 2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.660041 2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.660085 2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661613 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661638 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661651 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661664 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661681 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661693 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661709 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661721 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661735 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661746 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661777 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.661788 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.662707 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.662714 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.663975 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.664043 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.664663 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-212.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.666583 2576 server.go:1268] "Started kubelet" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.666635 2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.667355 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.667432 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.668601 2576 server.go:310] "Adding debug handlers to kubelet server" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.669466 2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.671400 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-212.ec2.internal.18c05339eb6a0561 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-212.ec2.internal,UID:ip-10-0-138-212.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-212.ec2.internal,},FirstTimestamp:2026-07-08 13:22:40.666551649 +0000 UTC m=+0.398389516,LastTimestamp:2026-07-08 13:22:40.666551649 +0000 UTC m=+0.398389516,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-212.ec2.internal,}" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.675364 2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.682368 2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.682377 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.682486 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683093 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683144 2576 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683159 2576 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683263 2576 reconstruct.go:97] "Volume reconstruction finished" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683273 2576 reconciler.go:29] "Reconciler: start to sync state" Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683311 2576 factory.go:55] Registering systemd factory Jul 08 13:22:41.055500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683328 2576 factory.go:223] Registration of the systemd container factory successfully Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683538 2576 factory.go:153] Registering CRI-O factory Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683550 2576 factory.go:223] Registration of the crio container factory successfully Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683590 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683608 2576 factory.go:103] Registering Raw factory Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.683624 2576 manager.go:1196] Started watching for new ooms in manager Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.684044 2576 manager.go:349] Starting recovery of all containers Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.694527 2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.694695 2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-212.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.694736 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7xq4" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.697564 2576 manager.go:354] Recovery completed Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.700409 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x7xq4" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.701474 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703178 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703197 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703207 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703929 2576 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703938 2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.703954 2576 state_mem.go:36] "Initialized new in-memory state store" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.704983 2576 policy_none.go:49] "None policy: Start" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.705001 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.705016 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.705963 2576 policy_none.go:47] "Start" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.706002 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.707843 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.707862 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.707871 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.760692 2576 manager.go:337] "Starting Device Plugin manager" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.760772 2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.760786 2576 server.go:87] "Starting device plugin registration server" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.761030 2576 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.761042 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.761315 2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.761533 2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.761541 2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.762004 2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.762048 2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.797029 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.798265 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.798300 2576 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.798320 2576 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.798353 2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.801020 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.861736 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.862734 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.862762 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.057448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.862772 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.862794 2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.873699 2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.873719 2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-212.ec2.internal\": node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.893391 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.901598 2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal"] Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.901649 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.902561 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.902587 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.902598 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904238 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904367 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904411 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904934 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904955 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.904965 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.905032 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.905074 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.905084 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.906303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.906326 2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.906965 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientMemory" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.906988 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.906998 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeHasSufficientPID" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.975868 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-212.ec2.internal\" not found" node="ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.980088 2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-212.ec2.internal\" not found" node="ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.984489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1788d5073bff9365a683c9a74d77f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-212.ec2.internal\" (UID: \"bd1788d5073bff9365a683c9a74d77f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.984510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:40.984528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.060048 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:40.993905 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085592 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1788d5073bff9365a683c9a74d77f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-212.ec2.internal\" (UID: \"bd1788d5073bff9365a683c9a74d77f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085619 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085639 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085686 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bd1788d5073bff9365a683c9a74d77f0-config\") pod \"kube-apiserver-proxy-ip-10-0-138-212.ec2.internal\" (UID: \"bd1788d5073bff9365a683c9a74d77f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085740 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.085767 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/78d5725b495bef7a2d9869f56ce08c81-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal\" (UID: \"78d5725b495bef7a2d9869f56ce08c81\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.185321 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:41.094686 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.195796 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:41.195561 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.311588 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.278773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.311588 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:41.283579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:41.311588 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:41.296463 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.446651 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:41.397035 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:41.574840 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:41.497462 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:42.835986 2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-212.ec2.internal\" not found" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.842676 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.851877 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.851988 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.852014 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.895578 2576 apiserver.go:52] "Watching apiserver" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.914284 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.915024 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj","openshift-cluster-node-tuning-operator/tuned-qr5cr","openshift-dns/node-resolver-cskqh","openshift-multus/multus-additional-cni-plugins-q6csc","openshift-multus/network-metrics-daemon-pjblf","openshift-network-diagnostics/network-check-target-njj9z","openshift-image-registry/node-ca-sgrrx","openshift-multus/multus-7f84b","openshift-network-operator/iptables-alerter-zlsmw","openshift-ovn-kubernetes/ovnkube-node-xp5tk","kube-system/global-pull-secret-syncer-8gljg","kube-system/konnectivity-agent-z7kws"] Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.916843 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.918001 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.918909 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.918932 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zb85s\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.919100 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.919150 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.919234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.919985 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.920094 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.920289 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-njqq9\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.920515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.920573 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.920574 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.921391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.921636 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kj4bg\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.921732 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:42.921776 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.922738 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxf8s\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.922763 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.922963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:42.923020 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.923048 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 13:22:42.929516 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.924306 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.924677 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.925512 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7f84b" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.926248 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.926538 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.926551 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.927446 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.927471 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rxggx\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.927894 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.929188 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrr5t\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.929201 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.929240 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 13:22:42.931218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.930611 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:42.932552 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.932122 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:42.932552 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.932513 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 13:22:42.932791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.932573 2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" Jul 08 13:22:42.932791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.932737 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:22:42.933129 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.932805 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.933373 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7qjwb\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.933444 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.933546 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:42.933625 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.934490 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.934524 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.934803 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal"] Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.934858 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.935406 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.935491 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.935622 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.935805 2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.935869 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.936017 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9qkf\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.936920 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5xcqm\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.936937 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.937115 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.940116 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 13:17:41 +0000 UTC" deadline="2028-02-26 15:39:31.777393992 +0000 UTC" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.940137 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14354h16m48.837259664s" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941371 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-systemd\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-device-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-modprobe-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941427 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbwn\" (UniqueName: \"kubernetes.io/projected/a1804049-94ed-4c4c-a122-c8ba0b2ca304-kube-api-access-dlbwn\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941466 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-var-lib-kubelet\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.043217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-registration-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvfs\" (UniqueName: \"kubernetes.io/projected/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kube-api-access-6qvfs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941520 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-host\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941540 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8stg\" (UniqueName: \"kubernetes.io/projected/5058507b-4126-4d96-9618-48ae609b35a3-kube-api-access-p8stg\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-os-release\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941601 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-sys-fs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941643 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysconfig\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-kubernetes\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-run\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz59v\" (UniqueName: \"kubernetes.io/projected/597e86f8-e61a-4172-a71b-77e69cbd0fe5-kube-api-access-qz59v\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-conf\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941727 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5058507b-4126-4d96-9618-48ae609b35a3-hosts-file\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.044601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941776 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-socket-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-sys\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941805 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-lib-modules\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058507b-4126-4d96-9618-48ae609b35a3-tmp-dir\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941837 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941854 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941869 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-etc-tuned\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-tmp\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941949 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nvk\" (UniqueName: \"kubernetes.io/projected/6da93733-7907-418e-b346-989b929c3c58-kube-api-access-f5nvk\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941973 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cnibin\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.941990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.943656 2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.943930 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal"] Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.956871 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p26hg" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:42.964500 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p26hg" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.024519 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043104 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-bin\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043131 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-kubelet\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.047849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043145 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9508464b-6316-489b-a555-5eb6b3304982-host\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043166 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-node-log\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-registration-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043242 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvfs\" (UniqueName: \"kubernetes.io/projected/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kube-api-access-6qvfs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043258 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-host\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043278 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-os-release\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043301 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-etc-kubernetes\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043323 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ced888d2-110a-4e00-a73c-92148e39a609-host-slash\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043370 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-host\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043375 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-registration-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043379 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-sys-fs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043398 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-os-release\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysconfig\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043439 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-sys-fs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043460 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-run\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysconfig\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-env-overrides\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.048956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043515 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-conf\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043531 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043549 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cnibin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043569 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9508464b-6316-489b-a555-5eb6b3304982-serviceca\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-run\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043587 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-konnectivity-ca\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043609 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-system-cni-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-socket-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043667 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-lib-modules\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043688 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058507b-4126-4d96-9618-48ae609b35a3-tmp-dir\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-conf\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-lib-modules\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043754 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-os-release\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cni-binary-copy\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-hostroot\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-systemd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.049863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-etc-tuned\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-tmp\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cnibin\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043908 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-agent-certs\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043927 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.043965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-socket-dir-parent\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cnibin\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044025 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-daemon-config\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044064 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-netns\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-etc-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044121 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-log-socket\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044146 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-modprobe-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044188 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044212 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-system-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ced888d2-110a-4e00-a73c-92148e39a609-iptables-alerter-script\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.052957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-modprobe-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044277 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbwn\" (UniqueName: \"kubernetes.io/projected/a1804049-94ed-4c4c-a122-c8ba0b2ca304-kube-api-access-dlbwn\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-var-lib-kubelet\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044332 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044363 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8stg\" (UniqueName: \"kubernetes.io/projected/5058507b-4126-4d96-9618-48ae609b35a3-kube-api-access-p8stg\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044407 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovn-node-metrics-cert\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044420 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-var-lib-kubelet\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044445 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-kubernetes\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz59v\" (UniqueName: \"kubernetes.io/projected/597e86f8-e61a-4172-a71b-77e69cbd0fe5-kube-api-access-qz59v\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044551 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-bin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044585 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-multus\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044611 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.044623 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044557 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-kubernetes\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.044702 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:43.544681244 +0000 UTC m=+2.038401034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5058507b-4126-4d96-9618-48ae609b35a3-hosts-file\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044745 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-conf-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044770 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-kubelet\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044792 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5058507b-4126-4d96-9618-48ae609b35a3-hosts-file\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044794 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-systemd-units\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-var-lib-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044856 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-script-lib\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-cni-binary-copy\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044881 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044906 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058507b-4126-4d96-9618-48ae609b35a3-tmp-dir\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/597e86f8-e61a-4172-a71b-77e69cbd0fe5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.044983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-sys\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.054931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045010 2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045028 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-socket-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045062 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-sys\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045082 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-config\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5hsg\" (UniqueName: \"kubernetes.io/projected/0b13e0f3-a04e-4bee-b465-3445c23c68b1-kube-api-access-n5hsg\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/597e86f8-e61a-4172-a71b-77e69cbd0fe5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045149 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nvk\" (UniqueName: \"kubernetes.io/projected/6da93733-7907-418e-b346-989b929c3c58-kube-api-access-f5nvk\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-netns\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-multus-certs\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045236 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-kubelet-config\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045257 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-dbus\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-ovn\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045306 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-netd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045336 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-systemd\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.056907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g767f\" (UniqueName: \"kubernetes.io/projected/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-kube-api-access-g767f\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045464 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-systemd\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045481 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6da93733-7907-418e-b346-989b929c3c58-etc-sysctl-d\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045486 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-928cd\" (UniqueName: \"kubernetes.io/projected/9508464b-6316-489b-a555-5eb6b3304982-kube-api-access-928cd\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpldb\" (UniqueName: \"kubernetes.io/projected/ced888d2-110a-4e00-a73c-92148e39a609-kube-api-access-rpldb\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045551 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-device-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-k8s-cni-cncf-io\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045600 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5e8f40f5-656d-4e70-aeac-95c51e8e3263-device-dir\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.045607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-slash\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.048969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-etc-tuned\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.050007 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6da93733-7907-418e-b346-989b929c3c58-tmp\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.055828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8stg\" (UniqueName: \"kubernetes.io/projected/5058507b-4126-4d96-9618-48ae609b35a3-kube-api-access-p8stg\") pod \"node-resolver-cskqh\" (UID: \"5058507b-4126-4d96-9618-48ae609b35a3\") " pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.056364 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz59v\" (UniqueName: \"kubernetes.io/projected/597e86f8-e61a-4172-a71b-77e69cbd0fe5-kube-api-access-qz59v\") pod \"multus-additional-cni-plugins-q6csc\" (UID: \"597e86f8-e61a-4172-a71b-77e69cbd0fe5\") " pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.056486 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbwn\" (UniqueName: \"kubernetes.io/projected/a1804049-94ed-4c4c-a122-c8ba0b2ca304-kube-api-access-dlbwn\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.057796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.057104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvfs\" (UniqueName: \"kubernetes.io/projected/5e8f40f5-656d-4e70-aeac-95c51e8e3263-kube-api-access-6qvfs\") pod \"aws-ebs-csi-driver-node-2l9fj\" (UID: \"5e8f40f5-656d-4e70-aeac-95c51e8e3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.060188 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.058014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nvk\" (UniqueName: \"kubernetes.io/projected/6da93733-7907-418e-b346-989b929c3c58-kube-api-access-f5nvk\") pod \"tuned-qr5cr\" (UID: \"6da93733-7907-418e-b346-989b929c3c58\") " pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145842 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-conf-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-kubelet\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-systemd-units\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-var-lib-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145939 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-script-lib\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-conf-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145961 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-config\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-kubelet\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145970 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-systemd-units\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.145983 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5hsg\" (UniqueName: \"kubernetes.io/projected/0b13e0f3-a04e-4bee-b465-3445c23c68b1-kube-api-access-n5hsg\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146004 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-netns\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-var-lib-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-multus-certs\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-kubelet-config\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-multus-certs\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146084 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-dbus\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146107 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-ovn\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.247049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146119 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-netns\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146124 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-kubelet-config\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146129 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-netd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-netd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146198 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146232 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-ovn\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146228 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146290 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146291 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146257 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c379c49d-db21-4f67-b2a8-b6c871b05a13-dbus\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g767f\" (UniqueName: \"kubernetes.io/projected/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-kube-api-access-g767f\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146335 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-928cd\" (UniqueName: \"kubernetes.io/projected/9508464b-6316-489b-a555-5eb6b3304982-kube-api-access-928cd\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146358 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpldb\" (UniqueName: \"kubernetes.io/projected/ced888d2-110a-4e00-a73c-92148e39a609-kube-api-access-rpldb\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-k8s-cni-cncf-io\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-slash\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146428 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-bin\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-kubelet\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.248728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146477 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-slash\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-run-k8s-cni-cncf-io\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146497 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-kubelet\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146501 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-cni-bin\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146513 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9508464b-6316-489b-a555-5eb6b3304982-host\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9508464b-6316-489b-a555-5eb6b3304982-host\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-node-log\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146575 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-etc-kubernetes\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146598 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-node-log\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ced888d2-110a-4e00-a73c-92148e39a609-host-slash\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146630 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-env-overrides\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-etc-kubernetes\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146646 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cnibin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146692 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9508464b-6316-489b-a555-5eb6b3304982-serviceca\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ced888d2-110a-4e00-a73c-92148e39a609-host-slash\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146719 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-konnectivity-ca\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146716 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cnibin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146752 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-os-release\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.249601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cni-binary-copy\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146801 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-hostroot\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146823 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-systemd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-os-release\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-agent-certs\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146871 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-hostroot\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146880 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-socket-dir-parent\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-daemon-config\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-netns\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.146975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-etc-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-log-socket\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147026 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-system-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ced888d2-110a-4e00-a73c-92148e39a609-iptables-alerter-script\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147073 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovn-node-metrics-cert\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-bin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147160 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9508464b-6316-489b-a555-5eb6b3304982-serviceca\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.250679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-multus\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-netns\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147270 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-systemd\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147296 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-script-lib\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-host-run-ovn-kubernetes\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147365 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-run-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-cni-binary-copy\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-konnectivity-ca\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-etc-openvswitch\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-env-overrides\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147526 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-multus\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147540 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-system-cni-dir\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147559 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-host-var-lib-cni-bin\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.147562 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147655 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b13e0f3-a04e-4bee-b465-3445c23c68b1-log-socket\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147566 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-socket-dir-parent\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.147698 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:43.647679569 +0000 UTC m=+2.141399337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/ced888d2-110a-4e00-a73c-92148e39a609-iptables-alerter-script\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.147955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-multus-daemon-config\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.148025 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovnkube-config\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.149496 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa-agent-certs\") pod \"konnectivity-agent-z7kws\" (UID: \"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa\") " pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.149687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b13e0f3-a04e-4bee-b465-3445c23c68b1-ovn-node-metrics-cert\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.155014 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.155034 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.155047 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.155122 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:22:43.655105733 +0000 UTC m=+2.148825541 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.155603 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.156401 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-928cd\" (UniqueName: \"kubernetes.io/projected/9508464b-6316-489b-a555-5eb6b3304982-kube-api-access-928cd\") pod \"node-ca-sgrrx\" (UID: \"9508464b-6316-489b-a555-5eb6b3304982\") " pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.157079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpldb\" (UniqueName: \"kubernetes.io/projected/ced888d2-110a-4e00-a73c-92148e39a609-kube-api-access-rpldb\") pod \"iptables-alerter-zlsmw\" (UID: \"ced888d2-110a-4e00-a73c-92148e39a609\") " pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.157672 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g767f\" (UniqueName: \"kubernetes.io/projected/0ae0a921-745f-411e-8ec7-33fcfb7a5bd3-kube-api-access-g767f\") pod \"multus-7f84b\" (UID: \"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3\") " pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.157706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5hsg\" (UniqueName: \"kubernetes.io/projected/0b13e0f3-a04e-4bee-b465-3445c23c68b1-kube-api-access-n5hsg\") pod \"ovnkube-node-xp5tk\" (UID: \"0b13e0f3-a04e-4bee-b465-3445c23c68b1\") " pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.170790 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.176048 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:22:43.252914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.241154 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" Jul 08 13:22:43.267415 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.267391 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" Jul 08 13:22:43.284896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.284871 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-cskqh" Jul 08 13:22:43.295833 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.295781 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q6csc" Jul 08 13:22:43.323967 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.323938 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-sgrrx" Jul 08 13:22:43.382477 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.382454 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7f84b" Jul 08 13:22:43.390920 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.390892 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae0a921_745f_411e_8ec7_33fcfb7a5bd3.slice/crio-ba2f5bb5710da5d8709fb1f0923026e9a106546d3de985344f2515c0c32dea1f WatchSource:0}: Error finding container ba2f5bb5710da5d8709fb1f0923026e9a106546d3de985344f2515c0c32dea1f: Status 404 returned error can't find the container with id ba2f5bb5710da5d8709fb1f0923026e9a106546d3de985344f2515c0c32dea1f Jul 08 13:22:43.391552 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.391527 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1788d5073bff9365a683c9a74d77f0.slice/crio-22cf81a5c14db3f94543337c70cef5e4fcb7c1f4994025782ed9a5eaf4e1df8e WatchSource:0}: Error finding container 22cf81a5c14db3f94543337c70cef5e4fcb7c1f4994025782ed9a5eaf4e1df8e: Status 404 returned error can't find the container with id 22cf81a5c14db3f94543337c70cef5e4fcb7c1f4994025782ed9a5eaf4e1df8e Jul 08 13:22:43.392208 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.392193 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d5725b495bef7a2d9869f56ce08c81.slice/crio-17b3647cf25ab18849095d8e334cc60a64dd4d01f035deb729b56b57d3cd61a3 WatchSource:0}: Error finding container 17b3647cf25ab18849095d8e334cc60a64dd4d01f035deb729b56b57d3cd61a3: Status 404 returned error can't find the container with id 17b3647cf25ab18849095d8e334cc60a64dd4d01f035deb729b56b57d3cd61a3 Jul 08 13:22:43.399526 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.399508 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:22:43.411463 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.411447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zlsmw" Jul 08 13:22:43.418954 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.418937 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced888d2_110a_4e00_a73c_92148e39a609.slice/crio-07cfafd095a4ae6007c7276d4b8dd0489e564d499118cc2b74f87970b1f54158 WatchSource:0}: Error finding container 07cfafd095a4ae6007c7276d4b8dd0489e564d499118cc2b74f87970b1f54158: Status 404 returned error can't find the container with id 07cfafd095a4ae6007c7276d4b8dd0489e564d499118cc2b74f87970b1f54158 Jul 08 13:22:43.461535 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.461514 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bb7d9e_09e8_4b37_a1eb_a1a6f7a910fa.slice/crio-55155fec0f93992d9ba0ee313cf09258099824f77146bfa25656b231d6912763 WatchSource:0}: Error finding container 55155fec0f93992d9ba0ee313cf09258099824f77146bfa25656b231d6912763: Status 404 returned error can't find the container with id 55155fec0f93992d9ba0ee313cf09258099824f77146bfa25656b231d6912763 Jul 08 13:22:43.504001 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.503979 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:43.550929 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.550860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:43.551037 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.550965 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:43.551037 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.551011 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:44.550997824 +0000 UTC m=+3.044717591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:43.628663 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.628628 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e8f40f5_656d_4e70_aeac_95c51e8e3263.slice/crio-f736b4d667644a9ca5679ad2b1c81458052454fe8610c4df2d63325c101a6712 WatchSource:0}: Error finding container f736b4d667644a9ca5679ad2b1c81458052454fe8610c4df2d63325c101a6712: Status 404 returned error can't find the container with id f736b4d667644a9ca5679ad2b1c81458052454fe8610c4df2d63325c101a6712 Jul 08 13:22:43.647032 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.647003 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da93733_7907_418e_b346_989b929c3c58.slice/crio-ac65a74b365b8a1cff4388341a3895e434531b4dfe42ce4ab245e619b1dd70e0 WatchSource:0}: Error finding container ac65a74b365b8a1cff4388341a3895e434531b4dfe42ce4ab245e619b1dd70e0: Status 404 returned error can't find the container with id ac65a74b365b8a1cff4388341a3895e434531b4dfe42ce4ab245e619b1dd70e0 Jul 08 13:22:43.651313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.651294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:43.651394 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.651383 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:43.651438 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.651429 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:44.651415576 +0000 UTC m=+3.145135344 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:43.706099 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.706071 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5058507b_4126_4d96_9618_48ae609b35a3.slice/crio-006a853a5c4e428beabf8ce57bff751b3b42c521670c040547c8560d21b9a226 WatchSource:0}: Error finding container 006a853a5c4e428beabf8ce57bff751b3b42c521670c040547c8560d21b9a226: Status 404 returned error can't find the container with id 006a853a5c4e428beabf8ce57bff751b3b42c521670c040547c8560d21b9a226 Jul 08 13:22:43.752386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.752362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:43.752505 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.752468 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:43.752505 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.752483 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:43.752505 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.752492 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:43.752597 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:43.752546 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:22:44.7525295 +0000 UTC m=+3.246249268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:43.792402 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.792377 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod597e86f8_e61a_4172_a71b_77e69cbd0fe5.slice/crio-5b492ed16c8c4b1f2d49f057352231d5c565eb66643eba4fecb99019b4e9f86e WatchSource:0}: Error finding container 5b492ed16c8c4b1f2d49f057352231d5c565eb66643eba4fecb99019b4e9f86e: Status 404 returned error can't find the container with id 5b492ed16c8c4b1f2d49f057352231d5c565eb66643eba4fecb99019b4e9f86e Jul 08 13:22:43.830427 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.830402 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b13e0f3_a04e_4bee_b465_3445c23c68b1.slice/crio-19831cbc8b66f6a162e40308f3de943c4dff3014822e694d3119df806fffb1eb WatchSource:0}: Error finding container 19831cbc8b66f6a162e40308f3de943c4dff3014822e694d3119df806fffb1eb: Status 404 returned error can't find the container with id 19831cbc8b66f6a162e40308f3de943c4dff3014822e694d3119df806fffb1eb Jul 08 13:22:43.845137 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:22:43.845112 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9508464b_6316_489b_a555_5eb6b3304982.slice/crio-a6e589308385039899899e815fbbc2096d20bc09d933bf35b9f70327aa2729db WatchSource:0}: Error finding container a6e589308385039899899e815fbbc2096d20bc09d933bf35b9f70327aa2729db: Status 404 returned error can't find the container with id a6e589308385039899899e815fbbc2096d20bc09d933bf35b9f70327aa2729db Jul 08 13:22:43.966287 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.966232 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 13:17:42 +0000 UTC" deadline="2028-02-18 19:32:16.244867589 +0000 UTC" Jul 08 13:22:43.966287 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:43.966268 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14166h9m32.278604379s" Jul 08 13:22:44.043216 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.043142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z7kws" event={"ID":"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa","Type":"ContainerStarted","Data":"55155fec0f93992d9ba0ee313cf09258099824f77146bfa25656b231d6912763"} Jul 08 13:22:44.045831 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.045804 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlsmw" event={"ID":"ced888d2-110a-4e00-a73c-92148e39a609","Type":"ContainerStarted","Data":"07cfafd095a4ae6007c7276d4b8dd0489e564d499118cc2b74f87970b1f54158"} Jul 08 13:22:44.048388 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.048333 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" event={"ID":"bd1788d5073bff9365a683c9a74d77f0","Type":"ContainerStarted","Data":"22cf81a5c14db3f94543337c70cef5e4fcb7c1f4994025782ed9a5eaf4e1df8e"} Jul 08 13:22:44.054606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.054132 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerStarted","Data":"5b492ed16c8c4b1f2d49f057352231d5c565eb66643eba4fecb99019b4e9f86e"} Jul 08 13:22:44.057229 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.057142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cskqh" event={"ID":"5058507b-4126-4d96-9618-48ae609b35a3","Type":"ContainerStarted","Data":"006a853a5c4e428beabf8ce57bff751b3b42c521670c040547c8560d21b9a226"} Jul 08 13:22:44.059185 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.059145 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" event={"ID":"5e8f40f5-656d-4e70-aeac-95c51e8e3263","Type":"ContainerStarted","Data":"f736b4d667644a9ca5679ad2b1c81458052454fe8610c4df2d63325c101a6712"} Jul 08 13:22:44.061212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.061142 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7f84b" event={"ID":"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3","Type":"ContainerStarted","Data":"ba2f5bb5710da5d8709fb1f0923026e9a106546d3de985344f2515c0c32dea1f"} Jul 08 13:22:44.062643 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.062620 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" event={"ID":"78d5725b495bef7a2d9869f56ce08c81","Type":"ContainerStarted","Data":"17b3647cf25ab18849095d8e334cc60a64dd4d01f035deb729b56b57d3cd61a3"} Jul 08 13:22:44.064041 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.064017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sgrrx" event={"ID":"9508464b-6316-489b-a555-5eb6b3304982","Type":"ContainerStarted","Data":"a6e589308385039899899e815fbbc2096d20bc09d933bf35b9f70327aa2729db"} Jul 08 13:22:44.065457 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.065432 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"19831cbc8b66f6a162e40308f3de943c4dff3014822e694d3119df806fffb1eb"} Jul 08 13:22:44.067386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.067357 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" event={"ID":"6da93733-7907-418e-b346-989b929c3c58","Type":"ContainerStarted","Data":"ac65a74b365b8a1cff4388341a3895e434531b4dfe42ce4ab245e619b1dd70e0"} Jul 08 13:22:44.444574 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.442997 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:44.560654 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.559942 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:44.560654 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.560185 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:44.560654 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.560254 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:46.560233273 +0000 UTC m=+5.053953057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:44.661342 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.660699 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:44.661342 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.660911 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:44.661342 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.660976 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:46.66095788 +0000 UTC m=+5.154677668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:44.762483 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.761813 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:44.762483 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.762003 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:44.762483 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.762026 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:44.762483 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.762038 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:44.762483 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:44.762092 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:22:46.762075378 +0000 UTC m=+5.255795165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:44.967077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.966905 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 13:17:42 +0000 UTC" deadline="2028-01-07 14:53:12.744099588 +0000 UTC" Jul 08 13:22:44.967077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:44.966965 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13153h30m27.777139049s" Jul 08 13:22:45.036753 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:45.036673 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:45.037015 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:45.036799 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:45.037015 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:45.036681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:45.037015 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:45.036681 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:45.037015 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:45.036901 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:45.037015 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:45.037004 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:45.665198 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:45.665155 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:22:46.578233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:46.577604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:46.578233 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.577760 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:46.578233 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.577825 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:50.5778065 +0000 UTC m=+9.071526272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:46.679115 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:46.678550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:46.679115 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.678718 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:46.679115 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.678780 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:50.678761326 +0000 UTC m=+9.172481099 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:46.780383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:46.779729 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:46.780383 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.779923 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:46.780383 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.779943 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:46.780383 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.779955 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:46.780383 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:46.780027 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:22:50.780006804 +0000 UTC m=+9.273726595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:47.037709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:47.037666 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:47.037886 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:47.037855 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:47.037955 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:47.037908 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:47.038022 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:47.038003 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:47.038093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:47.038046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:47.038149 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:47.038104 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:49.037109 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:49.037072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:49.037109 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:49.037103 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:49.037632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:49.037072 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:49.037632 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:49.037239 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:49.037632 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:49.037360 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:49.037632 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:49.037437 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:50.611570 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:50.611529 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:50.612021 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.611693 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:50.612021 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.611768 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:58.611748152 +0000 UTC m=+17.105467925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:50.712259 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:50.712220 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:50.712487 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.712367 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:50.712487 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.712446 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:22:58.712426378 +0000 UTC m=+17.206146161 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:50.813710 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:50.813554 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:50.813849 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.813715 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:50.813849 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.813743 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:50.813849 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.813757 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:50.813849 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:50.813817 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:22:58.813798268 +0000 UTC m=+17.307518040 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:51.037158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:51.037209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:51.037209 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:51.037309 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:51.037471 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:51.037549 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:51.037560 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:53.037428 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:53.037397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:53.037826 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:53.037533 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:53.037826 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:53.037397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:53.037826 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:53.037631 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:53.037826 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:53.037397 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:53.037826 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:53.037697 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:55.037048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:55.037003 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:55.037048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:55.037027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:55.037576 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:55.037027 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:55.037576 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:55.037158 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:55.037576 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:55.037219 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:55.037576 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:55.037297 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:57.037503 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:57.037470 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:57.037952 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:57.037468 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:57.037952 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:57.037595 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:57.037952 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:57.037478 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:57.037952 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:57.037672 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:22:57.037952 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:57.037737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:58.667769 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:58.667558 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:58.668219 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.667714 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:58.668219 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.667888 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:14.667872218 +0000 UTC m=+33.161591986 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:22:58.768635 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:58.768584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:58.768824 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.768729 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:58.768824 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.768808 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:14.768788023 +0000 UTC m=+33.262507795 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:22:58.869588 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:58.869549 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:58.869766 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.869703 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:22:58.869766 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.869721 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:22:58.869766 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.869734 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:58.869877 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:58.869797 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:23:14.869779811 +0000 UTC m=+33.363499581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:22:59.037266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:59.037184 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:22:59.037439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:59.037191 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:22:59.037439 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:59.037302 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:22:59.037439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:22:59.037319 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:22:59.037439 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:59.037401 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:22:59.037641 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:22:59.037479 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:01.037548 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:01.037514 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:01.037993 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:01.037515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:01.037993 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:01.037645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:01.037993 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:01.037515 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:01.037993 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:01.037731 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:01.037993 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:01.037810 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:03.037137 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:03.037092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:03.037582 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:03.037092 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:03.037582 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:03.037225 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:03.037582 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:03.037108 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:03.037582 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:03.037315 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:03.037582 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:03.037380 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:04.114126 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.113257 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" event={"ID":"6da93733-7907-418e-b346-989b929c3c58","Type":"ContainerStarted","Data":"d6b1c3174d0535a57024272f8541b54c4e090b80f76a30d6cd236b684f17ef8f"} Jul 08 13:23:04.126044 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.125757 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" event={"ID":"bd1788d5073bff9365a683c9a74d77f0","Type":"ContainerStarted","Data":"38da091d6e757b1391c257bf691635b7c1ee8148ad2505ddfe64d3b6b23b6751"} Jul 08 13:23:04.128837 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.128769 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7f84b" event={"ID":"0ae0a921-745f-411e-8ec7-33fcfb7a5bd3","Type":"ContainerStarted","Data":"4f3140a4871af68acf987539551301fe6202dc4b98204710678b0088381b33e1"} Jul 08 13:23:04.132444 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.132373 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"b37c2f63a757de730dc2a68ae2caa80a9039e070dcb0cfd70ca8e14bf2b7d393"} Jul 08 13:23:04.135033 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.134975 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qr5cr" podStartSLOduration=2.226056474 podStartE2EDuration="22.134957123s" totalImagesPullingTime="19.908900649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:04.13308986 +0000 UTC m=+22.626809671" watchObservedRunningTime="2026-07-08 13:23:04.134957123 +0000 UTC m=+22.628676913" Jul 08 13:23:04.153827 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.153345 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-7f84b" podStartSLOduration=1.9596428700000001 podStartE2EDuration="22.153328578s" totalImagesPullingTime="20.193685708s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:04.152806986 +0000 UTC m=+22.646526794" watchObservedRunningTime="2026-07-08 13:23:04.153328578 +0000 UTC m=+22.647048366" Jul 08 13:23:04.170499 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:04.170454 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-212.ec2.internal" podStartSLOduration=22.170439976 podStartE2EDuration="22.170439976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:04.170110739 +0000 UTC m=+22.663830528" watchObservedRunningTime="2026-07-08 13:23:04.170439976 +0000 UTC m=+22.664159764" Jul 08 13:23:05.037230 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.037020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:05.037404 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.037066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:05.037404 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:05.037307 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:05.037531 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:05.037396 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:05.037531 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.037066 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:05.037531 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:05.037492 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:05.136353 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.136282 2576 generic.go:356] "Generic (PLEG): container finished" podID="78d5725b495bef7a2d9869f56ce08c81" containerID="d085d561e3b2f9d91f564d9970974aa1e8418a1b621d160db3e1f909a58c154f" exitCode=0 Jul 08 13:23:05.137210 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.136370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" event={"ID":"78d5725b495bef7a2d9869f56ce08c81","Type":"ContainerDied","Data":"d085d561e3b2f9d91f564d9970974aa1e8418a1b621d160db3e1f909a58c154f"} Jul 08 13:23:05.138212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.138185 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-sgrrx" event={"ID":"9508464b-6316-489b-a555-5eb6b3304982","Type":"ContainerStarted","Data":"b5e4e9adbda217c700d9dc072eae157a6a3ce8f4fce9f8be3aec2831364cc87c"} Jul 08 13:23:05.141789 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.141766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"970b6c1e551522135f95a40412c9573e13d3c6a3ffc10e5a5c0cbee8ea301567"} Jul 08 13:23:05.141907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.141839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"bfa9d1120912903aa124585057a9d41ae8aed9079c466c0e9d47dd1966a4cf12"} Jul 08 13:23:05.141907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.141887 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"e5ef06c60d48609efe3c98860683be1bf8224b43a7d7028c6a04b795b663787a"} Jul 08 13:23:05.142019 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.141933 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"3b0b677c0474d63d8d45655056b44b210795363b2d606ae18c700c1a7e105400"} Jul 08 13:23:05.142019 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.141982 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"93a4cc5fa2d9449c667f2c64c062879d07873fc45c076af6c79c05911087e911"} Jul 08 13:23:05.143288 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.143268 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z7kws" event={"ID":"c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa","Type":"ContainerStarted","Data":"846ba5aaa413194761aef9d7209c214ff2cca2f16651b4707125c729c7cef3f0"} Jul 08 13:23:05.144868 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.144839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zlsmw" event={"ID":"ced888d2-110a-4e00-a73c-92148e39a609","Type":"ContainerStarted","Data":"f1461f15e37cc2a6a74010d91ed065db31a9428d3815bce5b85471b2c50f8f07"} Jul 08 13:23:05.146396 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.146373 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="ea16601c034fe4bbd76e1133a450dce73701c54aeb5f026f5d65eee87c32cdb9" exitCode=0 Jul 08 13:23:05.146494 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.146440 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"ea16601c034fe4bbd76e1133a450dce73701c54aeb5f026f5d65eee87c32cdb9"} Jul 08 13:23:05.147918 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.147883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-cskqh" event={"ID":"5058507b-4126-4d96-9618-48ae609b35a3","Type":"ContainerStarted","Data":"494378456a907c340664dcaee919a29e498c7f8fd4d33bb4df9c0da6c98503bc"} Jul 08 13:23:05.149501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.149479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" event={"ID":"5e8f40f5-656d-4e70-aeac-95c51e8e3263","Type":"ContainerStarted","Data":"a4dfcf25a17a525b6e30b135dd0f275499aca03d3598b3a570529a9c4a645c41"} Jul 08 13:23:05.165979 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.165926 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-sgrrx" podStartSLOduration=11.607043635 podStartE2EDuration="23.165909006s" totalImagesPullingTime="11.558865371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:05.165190186 +0000 UTC m=+23.658909974" watchObservedRunningTime="2026-07-08 13:23:05.165909006 +0000 UTC m=+23.659628794" Jul 08 13:23:05.178028 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.177992 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zlsmw" podStartSLOduration=3.062640017 podStartE2EDuration="23.177980147s" totalImagesPullingTime="20.11534013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:05.177818129 +0000 UTC m=+23.671537920" watchObservedRunningTime="2026-07-08 13:23:05.177980147 +0000 UTC m=+23.671699935" Jul 08 13:23:05.217269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.216852 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z7kws" podStartSLOduration=3.122261552 podStartE2EDuration="23.216832758s" totalImagesPullingTime="20.094571206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:05.201886037 +0000 UTC m=+23.695605852" watchObservedRunningTime="2026-07-08 13:23:05.216832758 +0000 UTC m=+23.710552551" Jul 08 13:23:05.217269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.217136 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-cskqh" podStartSLOduration=3.350018528 podStartE2EDuration="23.217127657s" totalImagesPullingTime="19.867109129s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:05.216349522 +0000 UTC m=+23.710069312" watchObservedRunningTime="2026-07-08 13:23:05.217127657 +0000 UTC m=+23.710847447" Jul 08 13:23:05.947674 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:05.947642 2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 13:23:06.013249 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.013129 2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T13:23:05.947669107Z","UUID":"766c160e-c015-40c4-8ebd-596398bf6b88","Handler":null,"Name":"","Endpoint":""} Jul 08 13:23:06.014559 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.014543 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 13:23:06.014648 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.014575 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 13:23:06.153812 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.153742 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" event={"ID":"5e8f40f5-656d-4e70-aeac-95c51e8e3263","Type":"ContainerStarted","Data":"acec6cec544ea5d783a77479d54c89e6e7e8f6818fda0c5f2428e7d38972e1d1"} Jul 08 13:23:06.156088 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.156049 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" event={"ID":"78d5725b495bef7a2d9869f56ce08c81","Type":"ContainerStarted","Data":"1baa7c8d2ecab91defe3786d41fc318a87fe3253548b19de66ff13eb033279c6"} Jul 08 13:23:06.171269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:06.171159 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-212.ec2.internal" podStartSLOduration=24.171141371 podStartE2EDuration="24.171141371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:06.170341634 +0000 UTC m=+24.664061440" watchObservedRunningTime="2026-07-08 13:23:06.171141371 +0000 UTC m=+24.664861162" Jul 08 13:23:07.037468 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:07.037395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:07.037468 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:07.037426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:07.037640 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:07.037395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:07.037640 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:07.037500 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:07.037640 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:07.037574 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:07.037746 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:07.037669 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:07.161634 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:07.161598 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"d81898aa3509cecbbf2e96bbe14705a369a3eb8082aeefa7408bc3092600c5bb"} Jul 08 13:23:08.165982 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:08.165927 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" event={"ID":"5e8f40f5-656d-4e70-aeac-95c51e8e3263","Type":"ContainerStarted","Data":"c88ac4dfba0f809c7d3ed1c9aac7f21dcedcb66413b6aa4e2f73a631034d1a8c"} Jul 08 13:23:08.320933 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:08.320904 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:23:08.321877 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:08.321859 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:23:08.336918 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:08.336877 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2l9fj" podStartSLOduration=3.879259356 podStartE2EDuration="26.336863728s" totalImagesPullingTime="22.457604372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:08.183189522 +0000 UTC m=+26.676909330" watchObservedRunningTime="2026-07-08 13:23:08.336863728 +0000 UTC m=+26.830583517" Jul 08 13:23:09.037233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.036902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:09.037233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.036902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:09.037233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.036959 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:09.037233 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:09.037236 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:09.037924 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:09.037346 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:09.037924 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:09.037468 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:09.172419 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.172372 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" event={"ID":"0b13e0f3-a04e-4bee-b465-3445c23c68b1","Type":"ContainerStarted","Data":"8f54fd30e1ea75b8be61d0aab9721cb3c1f2f726c5577ed70c893002d2836c89"} Jul 08 13:23:09.173278 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.172558 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:23:09.173439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.173420 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z7kws" Jul 08 13:23:09.204097 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:09.204046 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" podStartSLOduration=7.0429351 podStartE2EDuration="27.204032688s" totalImagesPullingTime="20.161097588s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:09.203870133 +0000 UTC m=+27.697589921" watchObservedRunningTime="2026-07-08 13:23:09.204032688 +0000 UTC m=+27.697752477" Jul 08 13:23:10.174785 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.174757 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:10.175214 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.174831 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:10.175214 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.174872 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:10.194795 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.194106 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:10.197225 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.197191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:10.996433 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.996189 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njj9z"] Jul 08 13:23:10.996610 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.996550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:10.996673 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:10.996648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:10.999046 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.999019 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gljg"] Jul 08 13:23:10.999207 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.999139 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:10.999281 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:10.999260 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:10.999759 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.999734 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pjblf"] Jul 08 13:23:10.999859 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:10.999847 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:10.999957 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:10.999937 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:13.037220 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:13.037183 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:13.037613 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:13.037281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:13.037613 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:13.037288 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:13.037613 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:13.037308 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:13.037613 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:13.037386 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:13.037613 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:13.037458 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:13.182117 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:13.182083 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="097cea43882eaa724735c671327507a1f5f4014084a9444861f85ccbd8f5c023" exitCode=0 Jul 08 13:23:13.182286 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:13.182130 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"097cea43882eaa724735c671327507a1f5f4014084a9444861f85ccbd8f5c023"} Jul 08 13:23:14.185673 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:14.185644 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="1795bf28780bd4e2a702ad46def8cfe644ab197a5f40ab87125b00c530c62106" exitCode=0 Jul 08 13:23:14.186046 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:14.185709 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"1795bf28780bd4e2a702ad46def8cfe644ab197a5f40ab87125b00c530c62106"} Jul 08 13:23:14.686601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:14.686560 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:14.686749 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.686729 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:23:14.686826 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.686809 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs podName:a1804049-94ed-4c4c-a122-c8ba0b2ca304 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:46.686787218 +0000 UTC m=+65.180506997 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs") pod "network-metrics-daemon-pjblf" (UID: "a1804049-94ed-4c4c-a122-c8ba0b2ca304") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 13:23:14.787450 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:14.787417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:14.787583 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.787521 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 13:23:14.787583 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.787575 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret podName:c379c49d-db21-4f67-b2a8-b6c871b05a13 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:46.787562058 +0000 UTC m=+65.281281825 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret") pod "global-pull-secret-syncer-8gljg" (UID: "c379c49d-db21-4f67-b2a8-b6c871b05a13") : object "kube-system"/"original-pull-secret" not registered Jul 08 13:23:14.887680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:14.887653 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:14.887863 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.887829 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 13:23:14.887863 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.887856 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 13:23:14.887990 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.887869 2576 projected.go:196] Error preparing data for projected volume kube-api-access-gcm9p for pod openshift-network-diagnostics/network-check-target-njj9z: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:23:14.887990 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:14.887928 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p podName:a5ebff38-dd48-450a-ae39-79bd1e92ca5f nodeName:}" failed. No retries permitted until 2026-07-08 13:23:46.887909053 +0000 UTC m=+65.381628832 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcm9p" (UniqueName: "kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p") pod "network-check-target-njj9z" (UID: "a5ebff38-dd48-450a-ae39-79bd1e92ca5f") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 13:23:15.036990 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:15.036953 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:15.037113 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:15.037057 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:15.037113 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:15.037065 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8gljg" podUID="c379c49d-db21-4f67-b2a8-b6c871b05a13" Jul 08 13:23:15.037233 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:15.037156 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pjblf" podUID="a1804049-94ed-4c4c-a122-c8ba0b2ca304" Jul 08 13:23:15.037233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:15.037064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:15.037296 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:15.037255 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njj9z" podUID="a5ebff38-dd48-450a-ae39-79bd1e92ca5f" Jul 08 13:23:15.189712 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:15.189674 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="152531c5069a7353b4eb9e2221e5465dfd9522f2c19067d301e8573be10935c6" exitCode=0 Jul 08 13:23:15.190063 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:15.189733 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"152531c5069a7353b4eb9e2221e5465dfd9522f2c19067d301e8573be10935c6"} Jul 08 13:23:16.798981 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.798743 2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-212.ec2.internal" event="NodeReady" Jul 08 13:23:16.799466 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.799109 2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 13:23:16.833460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.833421 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:23:16.847349 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.847315 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2"] Jul 08 13:23:16.847594 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.847496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:16.850591 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.850567 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 13:23:16.850718 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.850575 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dhbls\"" Jul 08 13:23:16.850718 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.850643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 13:23:16.850900 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.850740 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 13:23:16.864681 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.861869 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq"] Jul 08 13:23:16.864681 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.863924 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" Jul 08 13:23:16.865838 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.865816 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 13:23:16.868751 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.868328 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6j9wp\"" Jul 08 13:23:16.868751 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.868618 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.868751 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.868618 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.877531 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.877112 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-v4bwh"] Jul 08 13:23:16.877531 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.877239 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:16.879761 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.879731 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q5z89\"" Jul 08 13:23:16.879935 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.879917 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.880087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.880069 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 13:23:16.880305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.880099 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.889835 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.889807 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8"] Jul 08 13:23:16.890147 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.890129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:16.892431 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.892411 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7bmpq\"" Jul 08 13:23:16.892630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.892612 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 13:23:16.892777 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.892761 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.893058 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.893041 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 13:23:16.893116 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.893046 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.897595 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.897304 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 13:23:16.908369 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.908326 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx"] Jul 08 13:23:16.908494 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.908472 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:16.910792 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.910772 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 13:23:16.910909 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.910799 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 13:23:16.910975 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.910946 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vbgjp\"" Jul 08 13:23:16.911028 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.911006 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.911028 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.911019 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.924649 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924628 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2"] Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924687 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8"] Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924707 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq"] Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924731 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-v4bwh"] Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924743 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:16.924778 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.924755 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn"] Jul 08 13:23:16.926909 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.926885 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 13:23:16.927075 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.927057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 13:23:16.927277 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.927259 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.927340 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.927287 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cwv99\"" Jul 08 13:23:16.927340 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.927302 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.950783 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.950758 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-nplcg"] Jul 08 13:23:16.950914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.950883 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" Jul 08 13:23:16.953122 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.953102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.953248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.953158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6blql\"" Jul 08 13:23:16.953248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.953164 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.972615 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.970562 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rsqrk"] Jul 08 13:23:16.972615 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.971557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:16.986938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.986905 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 13:23:16.987124 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.986975 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qldzq\"" Jul 08 13:23:16.987238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.987219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 13:23:16.992997 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.992975 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4wnf4"] Jul 08 13:23:16.993415 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.993393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:16.995838 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.995820 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpm48\"" Jul 08 13:23:16.995984 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.995830 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 13:23:16.995984 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.995876 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 13:23:16.996216 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.996198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 13:23:16.999713 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.999690 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-trusted-ca\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:16.999828 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.999728 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:16.999828 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.999753 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n958\" (UniqueName: \"kubernetes.io/projected/002f971a-3e81-492b-b96e-cc929765f68b-kube-api-access-7n958\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:16.999828 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.999810 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgxcr\" (UniqueName: \"kubernetes.io/projected/4312d479-03e5-4b6f-97f3-4a99f77c09b2-kube-api-access-xgxcr\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:16.999995 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:16.999903 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/002f971a-3e81-492b-b96e-cc929765f68b-serving-cert\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.000052 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000005 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000052 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000035 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9e657f5-aaf8-4a08-a567-2346b505dfd7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.000151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000078 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9e657f5-aaf8-4a08-a567-2346b505dfd7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.000151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000136 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000273 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000217 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000273 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000347 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000281 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjngq\" (UniqueName: \"kubernetes.io/projected/a9e657f5-aaf8-4a08-a567-2346b505dfd7-kube-api-access-gjngq\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.000347 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000305 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-config\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.000347 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000335 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwnj\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000361 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:17.000471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000414 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.000569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.000528 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/33c0f971-0591-4a5a-9f95-acd98429c2ef-kube-api-access-6nwcd\") pod \"volume-data-source-validator-5d456fdbcd-gfnd2\" (UID: \"33c0f971-0591-4a5a-9f95-acd98429c2ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" Jul 08 13:23:17.013142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013122 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx"] Jul 08 13:23:17.013285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013160 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-nplcg"] Jul 08 13:23:17.013285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013194 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn"] Jul 08 13:23:17.013285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013213 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rsqrk"] Jul 08 13:23:17.013285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013227 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wnf4"] Jul 08 13:23:17.013285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.013273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.015622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.015602 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 13:23:17.015723 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.015604 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 13:23:17.015774 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.015726 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rlw5\"" Jul 08 13:23:17.037598 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.037571 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:17.037729 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.037696 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:17.037792 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.037765 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:17.040106 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.040088 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hs9z4\"" Jul 08 13:23:17.040247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.040120 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 13:23:17.040247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.040090 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 13:23:17.040247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.040090 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tktlz\"" Jul 08 13:23:17.101872 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.101839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9e657f5-aaf8-4a08-a567-2346b505dfd7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.101872 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.101875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.101894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.101911 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-config\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.102087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102048 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwnj\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ghf\" (UniqueName: \"kubernetes.io/projected/c624bc17-d20d-4de9-b708-48f183372910-kube-api-access-s7ghf\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102156 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6813cb3b-ee4c-405a-b290-7845d985063b-cert\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-trusted-ca\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102241 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102292 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43cc3d2a-0702-464a-a055-88824feaaa42-config-volume\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.102317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102315 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43cc3d2a-0702-464a-a055-88824feaaa42-tmp-dir\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102367 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9e657f5-aaf8-4a08-a567-2346b505dfd7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqhn\" (UniqueName: \"kubernetes.io/projected/43cc3d2a-0702-464a-a055-88824feaaa42-kube-api-access-gkqhn\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102424 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c624bc17-d20d-4de9-b708-48f183372910-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c624bc17-d20d-4de9-b708-48f183372910-config\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102479 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102509 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hclch\" (UniqueName: \"kubernetes.io/projected/6813cb3b-ee4c-405a-b290-7845d985063b-kube-api-access-hclch\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102538 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102547 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a9e657f5-aaf8-4a08-a567-2346b505dfd7-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.102622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjngq\" (UniqueName: \"kubernetes.io/projected/a9e657f5-aaf8-4a08-a567-2346b505dfd7-kube-api-access-gjngq\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102695 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-config\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4glf\" (UniqueName: \"kubernetes.io/projected/9476c443-e977-49f7-a302-97ea4cbdc9e8-kube-api-access-p4glf\") pod \"network-check-source-989f8b8dc-c7tkn\" (UID: \"9476c443-e977-49f7-a302-97ea4cbdc9e8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.102796 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.102856 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.102984 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:17.602924703 +0000 UTC m=+36.096644481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:17.103127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103006 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.103284 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.103298 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.103355 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:17.603340331 +0000 UTC m=+36.097060105 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103378 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/002f971a-3e81-492b-b96e-cc929765f68b-trusted-ca\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/33c0f971-0591-4a5a-9f95-acd98429c2ef-kube-api-access-6nwcd\") pod \"volume-data-source-validator-5d456fdbcd-gfnd2\" (UID: \"33c0f971-0591-4a5a-9f95-acd98429c2ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103407 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103433 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n958\" (UniqueName: \"kubernetes.io/projected/002f971a-3e81-492b-b96e-cc929765f68b-kube-api-access-7n958\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgxcr\" (UniqueName: \"kubernetes.io/projected/4312d479-03e5-4b6f-97f3-4a99f77c09b2-kube-api-access-xgxcr\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/002f971a-3e81-492b-b96e-cc929765f68b-serving-cert\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.103509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.103510 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.106473 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.106452 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a9e657f5-aaf8-4a08-a567-2346b505dfd7-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.106629 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.106612 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/002f971a-3e81-492b-b96e-cc929765f68b-serving-cert\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.106709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.106694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.106764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.106742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.112484 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.112461 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjngq\" (UniqueName: \"kubernetes.io/projected/a9e657f5-aaf8-4a08-a567-2346b505dfd7-kube-api-access-gjngq\") pod \"kube-storage-version-migrator-operator-846bf668f7-l7dm8\" (UID: \"a9e657f5-aaf8-4a08-a567-2346b505dfd7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.112985 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.112965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgxcr\" (UniqueName: \"kubernetes.io/projected/4312d479-03e5-4b6f-97f3-4a99f77c09b2-kube-api-access-xgxcr\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:17.113069 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.112983 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.113727 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.113702 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n958\" (UniqueName: \"kubernetes.io/projected/002f971a-3e81-492b-b96e-cc929765f68b-kube-api-access-7n958\") pod \"console-operator-68b757fb67-v4bwh\" (UID: \"002f971a-3e81-492b-b96e-cc929765f68b\") " pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.113878 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.113861 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nwcd\" (UniqueName: \"kubernetes.io/projected/33c0f971-0591-4a5a-9f95-acd98429c2ef-kube-api-access-6nwcd\") pod \"volume-data-source-validator-5d456fdbcd-gfnd2\" (UID: \"33c0f971-0591-4a5a-9f95-acd98429c2ef\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" Jul 08 13:23:17.114238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.114221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwnj\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.183368 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.183279 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" Jul 08 13:23:17.200606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.200574 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:17.204906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.204883 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43cc3d2a-0702-464a-a055-88824feaaa42-config-volume\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.204906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.204914 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43cc3d2a-0702-464a-a055-88824feaaa42-tmp-dir\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.205093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.204944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqhn\" (UniqueName: \"kubernetes.io/projected/43cc3d2a-0702-464a-a055-88824feaaa42-kube-api-access-gkqhn\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.205093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.204990 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c624bc17-d20d-4de9-b708-48f183372910-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.205093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205075 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c624bc17-d20d-4de9-b708-48f183372910-config\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.205266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.205266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hclch\" (UniqueName: \"kubernetes.io/projected/6813cb3b-ee4c-405a-b290-7845d985063b-kube-api-access-hclch\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.205266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.205266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205222 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4glf\" (UniqueName: \"kubernetes.io/projected/9476c443-e977-49f7-a302-97ea4cbdc9e8-kube-api-access-p4glf\") pod \"network-check-source-989f8b8dc-c7tkn\" (UID: \"9476c443-e977-49f7-a302-97ea4cbdc9e8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" Jul 08 13:23:17.205462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43cc3d2a-0702-464a-a055-88824feaaa42-tmp-dir\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.205462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.205462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205402 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ghf\" (UniqueName: \"kubernetes.io/projected/c624bc17-d20d-4de9-b708-48f183372910-kube-api-access-s7ghf\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.205462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205447 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6813cb3b-ee4c-405a-b290-7845d985063b-cert\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.205657 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.205604 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:17.205708 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.205659 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:17.70564246 +0000 UTC m=+36.199362238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:17.205708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205677 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/43cc3d2a-0702-464a-a055-88824feaaa42-config-volume\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.205819 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.205722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c624bc17-d20d-4de9-b708-48f183372910-config\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.205931 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.205913 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:17.205994 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.205971 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:17.705956939 +0000 UTC m=+36.199676721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:17.206113 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.206052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.207912 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.207889 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c624bc17-d20d-4de9-b708-48f183372910-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.208236 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.208219 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6813cb3b-ee4c-405a-b290-7845d985063b-cert\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.217719 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.217695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" Jul 08 13:23:17.218263 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.218215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4glf\" (UniqueName: \"kubernetes.io/projected/9476c443-e977-49f7-a302-97ea4cbdc9e8-kube-api-access-p4glf\") pod \"network-check-source-989f8b8dc-c7tkn\" (UID: \"9476c443-e977-49f7-a302-97ea4cbdc9e8\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" Jul 08 13:23:17.218456 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.218431 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ghf\" (UniqueName: \"kubernetes.io/projected/c624bc17-d20d-4de9-b708-48f183372910-kube-api-access-s7ghf\") pod \"service-ca-operator-5d46d9ffd4-pnljx\" (UID: \"c624bc17-d20d-4de9-b708-48f183372910\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.219016 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.218992 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hclch\" (UniqueName: \"kubernetes.io/projected/6813cb3b-ee4c-405a-b290-7845d985063b-kube-api-access-hclch\") pod \"ingress-canary-rsqrk\" (UID: \"6813cb3b-ee4c-405a-b290-7845d985063b\") " pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.219091 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.219052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqhn\" (UniqueName: \"kubernetes.io/projected/43cc3d2a-0702-464a-a055-88824feaaa42-kube-api-access-gkqhn\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.234088 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.234064 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" Jul 08 13:23:17.262953 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.262881 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" Jul 08 13:23:17.306801 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.306565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rsqrk" Jul 08 13:23:17.441359 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.441051 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-v4bwh"] Jul 08 13:23:17.445956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.445930 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2"] Jul 08 13:23:17.470596 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.470410 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8"] Jul 08 13:23:17.475440 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:17.474924 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e657f5_aaf8_4a08_a567_2346b505dfd7.slice/crio-64508b69879016010d4332977e1e33b3d5f80aba86f1bc70d60ac517fa38125f WatchSource:0}: Error finding container 64508b69879016010d4332977e1e33b3d5f80aba86f1bc70d60ac517fa38125f: Status 404 returned error can't find the container with id 64508b69879016010d4332977e1e33b3d5f80aba86f1bc70d60ac517fa38125f Jul 08 13:23:17.512024 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.511863 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn"] Jul 08 13:23:17.517022 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:17.516995 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9476c443_e977_49f7_a302_97ea4cbdc9e8.slice/crio-b2b3ef0116a26cb2c903b1001f90ed21d6ee6af424113a06c4b2f80a8a8048e8 WatchSource:0}: Error finding container b2b3ef0116a26cb2c903b1001f90ed21d6ee6af424113a06c4b2f80a8a8048e8: Status 404 returned error can't find the container with id b2b3ef0116a26cb2c903b1001f90ed21d6ee6af424113a06c4b2f80a8a8048e8 Jul 08 13:23:17.529551 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.528382 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx"] Jul 08 13:23:17.541920 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.541895 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rsqrk"] Jul 08 13:23:17.545206 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:17.545167 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6813cb3b_ee4c_405a_b290_7845d985063b.slice/crio-0620a2fbbb451e72be4cbffc006e13485c7e84cdcb2b133e1c4ff255364784ad WatchSource:0}: Error finding container 0620a2fbbb451e72be4cbffc006e13485c7e84cdcb2b133e1c4ff255364784ad: Status 404 returned error can't find the container with id 0620a2fbbb451e72be4cbffc006e13485c7e84cdcb2b133e1c4ff255364784ad Jul 08 13:23:17.609587 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.609477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:17.609587 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.609512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:17.609788 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.609621 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:17.609788 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.609631 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:17.609788 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.609633 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:17.609788 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.609689 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:18.609676584 +0000 UTC m=+37.103396351 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:17.609788 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.609703 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:18.609697533 +0000 UTC m=+37.103417300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:17.710678 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.710635 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:17.710854 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.710771 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:17.710854 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.710803 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:17.710940 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.710884 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:17.710940 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.710884 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:18.710863105 +0000 UTC m=+37.204582875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:17.711016 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:17.710941 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:18.710926319 +0000 UTC m=+37.204646086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:17.895386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.895338 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:17.973683 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:17.973632 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:18.074126 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.074061 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:18.175238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.175144 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:18.197948 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.197883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" event={"ID":"33c0f971-0591-4a5a-9f95-acd98429c2ef","Type":"ContainerStarted","Data":"30e59f994557784862e0176cb2c76f08c37ef9011ae1964caea213b06d2b1181"} Jul 08 13:23:18.200839 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.200810 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" event={"ID":"c624bc17-d20d-4de9-b708-48f183372910","Type":"ContainerStarted","Data":"eaa2f95be8c245c8de04a4d1a6477ee2dd5746072ecc58f16fd0dcc09105c131"} Jul 08 13:23:18.203123 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.203036 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" event={"ID":"9476c443-e977-49f7-a302-97ea4cbdc9e8","Type":"ContainerStarted","Data":"b2b3ef0116a26cb2c903b1001f90ed21d6ee6af424113a06c4b2f80a8a8048e8"} Jul 08 13:23:18.210050 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.209984 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" event={"ID":"002f971a-3e81-492b-b96e-cc929765f68b","Type":"ContainerStarted","Data":"72fd1a033763d0c5a51740d04c0090ab9dcb0371f5f2b5fe59c8ee8def1ecdcf"} Jul 08 13:23:18.212928 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.212855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rsqrk" event={"ID":"6813cb3b-ee4c-405a-b290-7845d985063b","Type":"ContainerStarted","Data":"0620a2fbbb451e72be4cbffc006e13485c7e84cdcb2b133e1c4ff255364784ad"} Jul 08 13:23:18.216968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.216900 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" event={"ID":"a9e657f5-aaf8-4a08-a567-2346b505dfd7","Type":"ContainerStarted","Data":"64508b69879016010d4332977e1e33b3d5f80aba86f1bc70d60ac517fa38125f"} Jul 08 13:23:18.372002 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.371941 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:18.428482 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.428323 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:18.619285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.619158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:18.619285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.619225 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:18.619531 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.619369 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:18.619531 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.619384 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:18.619531 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.619441 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:20.619422711 +0000 UTC m=+39.113142481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:18.619863 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.619841 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:18.619932 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.619901 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:20.619885955 +0000 UTC m=+39.113605728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:18.720518 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.720430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:18.720690 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:18.720526 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:18.720690 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.720668 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:18.720802 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.720726 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:20.720709032 +0000 UTC m=+39.214428803 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:18.721125 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.721104 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:18.721234 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:18.721163 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:20.721146028 +0000 UTC m=+39.214865802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:19.172525 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:19.172464 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:19.232467 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:19.232416 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:19.568286 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:19.568166 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:19.633196 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:19.633022 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:20.024621 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.024578 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:20.078088 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.078053 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:20.637686 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.637656 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.637697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.637796 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.637808 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.637850 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:24.637837578 +0000 UTC m=+43.131557345 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.637796 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:20.638107 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.637917 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:24.637901958 +0000 UTC m=+43.131621725 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:20.738580 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.738545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:20.738750 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:20.738615 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:20.738750 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.738718 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:20.738750 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.738742 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:20.738893 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.738804 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:24.738779418 +0000 UTC m=+43.232499206 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:20.738893 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:20.738827 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:24.738816262 +0000 UTC m=+43.232536032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:21.998741 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:21.998689 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:22.053921 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:22.053882 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:24.672012 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:24.671974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:24.672081 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.672149 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.672198 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.672162 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.672267 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:32.672245887 +0000 UTC m=+51.165965675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:24.672485 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.672309 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:32.672294264 +0000 UTC m=+51.166014030 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:24.773136 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:24.773099 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:24.773314 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:24.773216 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:24.773314 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.773254 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:24.773314 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.773314 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:32.773299089 +0000 UTC m=+51.267018862 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:24.773444 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.773328 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:24.773444 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:24.773370 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:32.773356587 +0000 UTC m=+51.267076353 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:27.242505 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.242463 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" event={"ID":"9476c443-e977-49f7-a302-97ea4cbdc9e8","Type":"ContainerStarted","Data":"ed7614754e08119eb5cefe49ea4485dd19b9d28ae6d2a45cca6a80e56f73e4f9"} Jul 08 13:23:27.244200 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.244157 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/0.log" Jul 08 13:23:27.244342 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.244216 2576 generic.go:356] "Generic (PLEG): container finished" podID="002f971a-3e81-492b-b96e-cc929765f68b" containerID="41a5f9819e40d745bc95af6c585c77ae6358e073755c403a90dc23873a51b91a" exitCode=255 Jul 08 13:23:27.244342 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.244306 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" event={"ID":"002f971a-3e81-492b-b96e-cc929765f68b","Type":"ContainerDied","Data":"41a5f9819e40d745bc95af6c585c77ae6358e073755c403a90dc23873a51b91a"} Jul 08 13:23:27.244490 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.244463 2576 scope.go:117] "RemoveContainer" containerID="41a5f9819e40d745bc95af6c585c77ae6358e073755c403a90dc23873a51b91a" Jul 08 13:23:27.247135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.247036 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="7b67f3e1c3ce4c4209a9c4b4972717661be0ecec9e96191cb0785e570ae97939" exitCode=0 Jul 08 13:23:27.247135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.247085 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"7b67f3e1c3ce4c4209a9c4b4972717661be0ecec9e96191cb0785e570ae97939"} Jul 08 13:23:27.248946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.248924 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rsqrk" event={"ID":"6813cb3b-ee4c-405a-b290-7845d985063b","Type":"ContainerStarted","Data":"109ab64a80e982e79b357cc31b6c9811c7ab7c1afaf63182de74823870b1166e"} Jul 08 13:23:27.253769 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.253695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" event={"ID":"a9e657f5-aaf8-4a08-a567-2346b505dfd7","Type":"ContainerStarted","Data":"9e683825d939075549b6554576b771354e39f48b0c94682c4375c91ee7f237b0"} Jul 08 13:23:27.255629 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.255607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" event={"ID":"33c0f971-0591-4a5a-9f95-acd98429c2ef","Type":"ContainerStarted","Data":"9d597a3a3b76b6b463252a3cd0e506ae55a9399dea0e5f5c6a195cf957f75888"} Jul 08 13:23:27.258688 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.258667 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" event={"ID":"c624bc17-d20d-4de9-b708-48f183372910","Type":"ContainerStarted","Data":"c7c82c47863df331a320ba82ab9e114027aa7d1d053c3aa9da09d75419b54b9e"} Jul 08 13:23:27.273972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.273932 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-c7tkn" podStartSLOduration=29.383115447 podStartE2EDuration="38.273916341s" totalImagesPullingTime="8.890800894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:27.256582888 +0000 UTC m=+45.750302677" watchObservedRunningTime="2026-07-08 13:23:27.273916341 +0000 UTC m=+45.767636124" Jul 08 13:23:27.274260 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.274238 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rsqrk" podStartSLOduration=2.421322896 podStartE2EDuration="11.274232273s" totalImagesPullingTime="8.852909377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:27.273308409 +0000 UTC m=+45.767028198" watchObservedRunningTime="2026-07-08 13:23:27.274232273 +0000 UTC m=+45.767952062" Jul 08 13:23:27.313234 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.313002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-gfnd2" podStartSLOduration=35.610159397 podStartE2EDuration="44.312982925s" totalImagesPullingTime="8.702823528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:27.312958137 +0000 UTC m=+45.806677927" watchObservedRunningTime="2026-07-08 13:23:27.312982925 +0000 UTC m=+45.806702716" Jul 08 13:23:27.343085 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.342731 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" podStartSLOduration=30.421155089 podStartE2EDuration="39.342710639s" totalImagesPullingTime="8.92155555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:27.342203963 +0000 UTC m=+45.835923776" watchObservedRunningTime="2026-07-08 13:23:27.342710639 +0000 UTC m=+45.836430430" Jul 08 13:23:27.358129 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.358052 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" podStartSLOduration=30.490582691 podStartE2EDuration="39.358030589s" totalImagesPullingTime="8.867447898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:27.357320327 +0000 UTC m=+45.851040116" watchObservedRunningTime="2026-07-08 13:23:27.358030589 +0000 UTC m=+45.851750378" Jul 08 13:23:27.390550 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.390521 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:27.892062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.892029 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:27.946231 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:27.946156 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.013544 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.013505 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.114656 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.114618 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.266152 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.266072 2576 generic.go:356] "Generic (PLEG): container finished" podID="597e86f8-e61a-4172-a71b-77e69cbd0fe5" containerID="ea33e5d7d49ca7cf3be4cf655c0fb0e09b2a0628a1db4be233ed7cfba334c9c4" exitCode=0 Jul 08 13:23:28.266570 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.266160 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerDied","Data":"ea33e5d7d49ca7cf3be4cf655c0fb0e09b2a0628a1db4be233ed7cfba334c9c4"} Jul 08 13:23:28.267809 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.267788 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:23:28.268195 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.268165 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/0.log" Jul 08 13:23:28.268257 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.268214 2576 generic.go:356] "Generic (PLEG): container finished" podID="002f971a-3e81-492b-b96e-cc929765f68b" containerID="2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2" exitCode=255 Jul 08 13:23:28.268313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.268294 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" event={"ID":"002f971a-3e81-492b-b96e-cc929765f68b","Type":"ContainerDied","Data":"2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2"} Jul 08 13:23:28.268354 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.268343 2576 scope.go:117] "RemoveContainer" containerID="41a5f9819e40d745bc95af6c585c77ae6358e073755c403a90dc23873a51b91a" Jul 08 13:23:28.268799 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.268563 2576 scope.go:117] "RemoveContainer" containerID="2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2" Jul 08 13:23:28.268799 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:28.268745 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-v4bwh_openshift-console-operator(002f971a-3e81-492b-b96e-cc929765f68b)\"" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" podUID="002f971a-3e81-492b-b96e-cc929765f68b" Jul 08 13:23:28.329054 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.329015 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.383046 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.383008 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.618919 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.618813 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:28.708721 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:28.708684 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:29.010349 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.010316 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89"] Jul 08 13:23:29.029825 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.029797 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89"] Jul 08 13:23:29.029947 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.029886 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" Jul 08 13:23:29.035184 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.035151 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 13:23:29.036167 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.036145 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5pd96\"" Jul 08 13:23:29.036259 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.036145 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 13:23:29.099995 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.099961 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:29.119354 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.119330 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gv7h\" (UniqueName: \"kubernetes.io/projected/f112cec5-a7e4-42d2-9c08-34c0dca62bef-kube-api-access-4gv7h\") pod \"migrator-d57d58cc9-w5h89\" (UID: \"f112cec5-a7e4-42d2-9c08-34c0dca62bef\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" Jul 08 13:23:29.192224 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.192189 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:29.220382 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.220354 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gv7h\" (UniqueName: \"kubernetes.io/projected/f112cec5-a7e4-42d2-9c08-34c0dca62bef-kube-api-access-4gv7h\") pod \"migrator-d57d58cc9-w5h89\" (UID: \"f112cec5-a7e4-42d2-9c08-34c0dca62bef\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" Jul 08 13:23:29.229494 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.229464 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gv7h\" (UniqueName: \"kubernetes.io/projected/f112cec5-a7e4-42d2-9c08-34c0dca62bef-kube-api-access-4gv7h\") pod \"migrator-d57d58cc9-w5h89\" (UID: \"f112cec5-a7e4-42d2-9c08-34c0dca62bef\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" Jul 08 13:23:29.273488 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.273424 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:23:29.273834 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.273787 2576 scope.go:117] "RemoveContainer" containerID="2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2" Jul 08 13:23:29.273952 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:29.273931 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-v4bwh_openshift-console-operator(002f971a-3e81-492b-b96e-cc929765f68b)\"" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" podUID="002f971a-3e81-492b-b96e-cc929765f68b" Jul 08 13:23:29.276678 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.276643 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q6csc" event={"ID":"597e86f8-e61a-4172-a71b-77e69cbd0fe5","Type":"ContainerStarted","Data":"a95544b8cc50f79cf98f51ba2e11c2701f5db52441cbac428c3f81a8b36c8fc2"} Jul 08 13:23:29.314735 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.314685 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q6csc" podStartSLOduration=8.001658717 podStartE2EDuration="47.314671707s" totalImagesPullingTime="39.31301299s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:29.313017412 +0000 UTC m=+47.806737232" watchObservedRunningTime="2026-07-08 13:23:29.314671707 +0000 UTC m=+47.808391495" Jul 08 13:23:29.339227 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.339198 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" Jul 08 13:23:29.435838 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.435793 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:29.480246 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.480215 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89"] Jul 08 13:23:29.483816 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:29.483789 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf112cec5_a7e4_42d2_9c08_34c0dca62bef.slice/crio-f33418bc88a2e964e04fbdcc8e3214bb599ac955022942e6cd9a822fb21ac86c WatchSource:0}: Error finding container f33418bc88a2e964e04fbdcc8e3214bb599ac955022942e6cd9a822fb21ac86c: Status 404 returned error can't find the container with id f33418bc88a2e964e04fbdcc8e3214bb599ac955022942e6cd9a822fb21ac86c Jul 08 13:23:29.536210 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:29.536120 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:30.195863 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.195834 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q2djv"] Jul 08 13:23:30.208117 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.208089 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q2djv"] Jul 08 13:23:30.208289 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.208219 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.211033 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.210958 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 13:23:30.211152 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.211034 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 13:23:30.212051 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.211964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 13:23:30.212051 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.212011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 13:23:30.212051 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.212043 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zs278\"" Jul 08 13:23:30.228102 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.228077 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f3836-5f53-48e3-bd6b-69182faafec1-signing-cabundle\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.228294 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.228264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7clm\" (UniqueName: \"kubernetes.io/projected/309f3836-5f53-48e3-bd6b-69182faafec1-kube-api-access-j7clm\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.228368 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.228319 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f3836-5f53-48e3-bd6b-69182faafec1-signing-key\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.232795 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.232715 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:30.282641 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.282607 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" event={"ID":"f112cec5-a7e4-42d2-9c08-34c0dca62bef","Type":"ContainerStarted","Data":"f33418bc88a2e964e04fbdcc8e3214bb599ac955022942e6cd9a822fb21ac86c"} Jul 08 13:23:30.329468 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.329430 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7clm\" (UniqueName: \"kubernetes.io/projected/309f3836-5f53-48e3-bd6b-69182faafec1-kube-api-access-j7clm\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.329468 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.329471 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f3836-5f53-48e3-bd6b-69182faafec1-signing-key\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.329693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.329671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f3836-5f53-48e3-bd6b-69182faafec1-signing-cabundle\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.330341 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.330318 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/309f3836-5f53-48e3-bd6b-69182faafec1-signing-cabundle\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.331805 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.331786 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/309f3836-5f53-48e3-bd6b-69182faafec1-signing-key\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.337921 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.337902 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7clm\" (UniqueName: \"kubernetes.io/projected/309f3836-5f53-48e3-bd6b-69182faafec1-kube-api-access-j7clm\") pod \"service-ca-758f599c5-q2djv\" (UID: \"309f3836-5f53-48e3-bd6b-69182faafec1\") " pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.519758 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.519668 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-q2djv" Jul 08 13:23:30.665998 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:30.665969 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-q2djv"] Jul 08 13:23:30.669075 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:30.669048 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod309f3836_5f53_48e3_bd6b_69182faafec1.slice/crio-8e47c336fa738a70b74c25c58666c7ff99923b4db85342d446aefcf33f97a05d WatchSource:0}: Error finding container 8e47c336fa738a70b74c25c58666c7ff99923b4db85342d446aefcf33f97a05d: Status 404 returned error can't find the container with id 8e47c336fa738a70b74c25c58666c7ff99923b4db85342d446aefcf33f97a05d Jul 08 13:23:31.099283 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:31.099240 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:31.287524 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:31.287442 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q2djv" event={"ID":"309f3836-5f53-48e3-bd6b-69182faafec1","Type":"ContainerStarted","Data":"0ca8afe79aa1a6987b0f1b4c6090fa0f3c54beaeca87ec86c19d9963b566da94"} Jul 08 13:23:31.287524 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:31.287498 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-q2djv" event={"ID":"309f3836-5f53-48e3-bd6b-69182faafec1","Type":"ContainerStarted","Data":"8e47c336fa738a70b74c25c58666c7ff99923b4db85342d446aefcf33f97a05d"} Jul 08 13:23:31.304318 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:31.304269 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-q2djv" podStartSLOduration=1.304255031 podStartE2EDuration="1.304255031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:31.303226543 +0000 UTC m=+49.796946356" watchObservedRunningTime="2026-07-08 13:23:31.304255031 +0000 UTC m=+49.797974820" Jul 08 13:23:32.292353 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.292312 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" event={"ID":"f112cec5-a7e4-42d2-9c08-34c0dca62bef","Type":"ContainerStarted","Data":"6c371a95d901279b81a5a3c60c8829dc38f648ff07bf13a0c5ad53050a2d1a0c"} Jul 08 13:23:32.292755 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.292420 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" event={"ID":"f112cec5-a7e4-42d2-9c08-34c0dca62bef","Type":"ContainerStarted","Data":"94c652785aef0d6da092a59fcdd5bae1a878b6884a9514de6ee28af6275c12ee"} Jul 08 13:23:32.310885 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.310837 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-w5h89" podStartSLOduration=2.438465844 podStartE2EDuration="4.310824751s" totalImagesPullingTime="1.872358907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:32.309929775 +0000 UTC m=+50.803649588" watchObservedRunningTime="2026-07-08 13:23:32.310824751 +0000 UTC m=+50.804544540" Jul 08 13:23:32.752800 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.752764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:32.752800 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.752807 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:32.753023 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.752903 2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 13:23:32.753023 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.752915 2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cf9c5fc8d-7drdq: secret "image-registry-tls" not found Jul 08 13:23:32.753023 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.752931 2576 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 13:23:32.753023 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.752967 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls podName:5bb0e178-c819-4862-bced-3175df56130e nodeName:}" failed. No retries permitted until 2026-07-08 13:23:48.752952214 +0000 UTC m=+67.246671981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls") pod "image-registry-7cf9c5fc8d-7drdq" (UID: "5bb0e178-c819-4862-bced-3175df56130e") : secret "image-registry-tls" not found Jul 08 13:23:32.753023 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.753009 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls podName:4312d479-03e5-4b6f-97f3-4a99f77c09b2 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:48.752991963 +0000 UTC m=+67.246711735 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-bhdbq" (UID: "4312d479-03e5-4b6f-97f3-4a99f77c09b2") : secret "samples-operator-tls" not found Jul 08 13:23:32.853790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.853757 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:32.853950 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:32.853828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:32.853950 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.853932 2576 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 13:23:32.854022 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.853986 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert podName:ac17ea6d-b1f3-498a-a5d1-ee940a61ec70 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:48.853973224 +0000 UTC m=+67.347692990 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-nplcg" (UID: "ac17ea6d-b1f3-498a-a5d1-ee940a61ec70") : secret "networking-console-plugin-cert" not found Jul 08 13:23:32.854062 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.853929 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 13:23:32.854140 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:32.854080 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls podName:43cc3d2a-0702-464a-a055-88824feaaa42 nodeName:}" failed. No retries permitted until 2026-07-08 13:23:48.854064665 +0000 UTC m=+67.347784450 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls") pod "dns-default-4wnf4" (UID: "43cc3d2a-0702-464a-a055-88824feaaa42") : secret "dns-default-metrics-tls" not found Jul 08 13:23:33.200712 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.200676 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.300591 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.300562 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.401729 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.401701 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.502755 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.502658 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.607664 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.607626 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.914893 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.914851 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:33.991632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:33.991597 2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" limit=1024 Jul 08 13:23:37.200906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:37.200856 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:37.200906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:37.200924 2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:37.201418 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:37.201266 2576 scope.go:117] "RemoveContainer" containerID="2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2" Jul 08 13:23:37.201418 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:23:37.201399 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-v4bwh_openshift-console-operator(002f971a-3e81-492b-b96e-cc929765f68b)\"" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" podUID="002f971a-3e81-492b-b96e-cc929765f68b" Jul 08 13:23:42.189513 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:42.189480 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xp5tk" Jul 08 13:23:46.763810 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.763773 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:46.766259 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.766230 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 13:23:46.776549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.776528 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a1804049-94ed-4c4c-a122-c8ba0b2ca304-metrics-certs\") pod \"network-metrics-daemon-pjblf\" (UID: \"a1804049-94ed-4c4c-a122-c8ba0b2ca304\") " pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:46.864285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.864251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:46.866852 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.866832 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 13:23:46.876955 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.876933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c379c49d-db21-4f67-b2a8-b6c871b05a13-original-pull-secret\") pod \"global-pull-secret-syncer-8gljg\" (UID: \"c379c49d-db21-4f67-b2a8-b6c871b05a13\") " pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:46.964615 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.964584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:46.967014 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:46.966993 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcm9p\" (UniqueName: \"kubernetes.io/projected/a5ebff38-dd48-450a-ae39-79bd1e92ca5f-kube-api-access-gcm9p\") pod \"network-check-target-njj9z\" (UID: \"a5ebff38-dd48-450a-ae39-79bd1e92ca5f\") " pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:47.050868 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.050791 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hs9z4\"" Jul 08 13:23:47.053914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.053901 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8gljg" Jul 08 13:23:47.058998 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.058975 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:47.060049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.060025 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tktlz\"" Jul 08 13:23:47.068248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.068225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pjblf" Jul 08 13:23:47.214691 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.214637 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njj9z"] Jul 08 13:23:47.218271 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:47.218243 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ebff38_dd48_450a_ae39_79bd1e92ca5f.slice/crio-6853bc8316d1c9df4a88b83dfd1176cc18cff9a5b72dad21551a0e1b68390887 WatchSource:0}: Error finding container 6853bc8316d1c9df4a88b83dfd1176cc18cff9a5b72dad21551a0e1b68390887: Status 404 returned error can't find the container with id 6853bc8316d1c9df4a88b83dfd1176cc18cff9a5b72dad21551a0e1b68390887 Jul 08 13:23:47.339152 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.339065 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njj9z" event={"ID":"a5ebff38-dd48-450a-ae39-79bd1e92ca5f","Type":"ContainerStarted","Data":"3fe8f511145c23d9a31a1a044a5e50bdef78d1f7e946062c24362a9e7dfb5931"} Jul 08 13:23:47.339320 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.339150 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njj9z" event={"ID":"a5ebff38-dd48-450a-ae39-79bd1e92ca5f","Type":"ContainerStarted","Data":"6853bc8316d1c9df4a88b83dfd1176cc18cff9a5b72dad21551a0e1b68390887"} Jul 08 13:23:47.339320 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.339287 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:23:47.353594 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.353544 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njj9z" podStartSLOduration=65.353530958 podStartE2EDuration="1m5.353530958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:47.353154976 +0000 UTC m=+65.846874786" watchObservedRunningTime="2026-07-08 13:23:47.353530958 +0000 UTC m=+65.847250725" Jul 08 13:23:47.437916 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.437881 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8gljg"] Jul 08 13:23:47.440424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:47.440399 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pjblf"] Jul 08 13:23:47.440971 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:47.440926 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc379c49d_db21_4f67_b2a8_b6c871b05a13.slice/crio-8a5b0ea75a5a1e7c4e1daf619db65d0b5ea09804d7e617f7ce0767272035c6f6 WatchSource:0}: Error finding container 8a5b0ea75a5a1e7c4e1daf619db65d0b5ea09804d7e617f7ce0767272035c6f6: Status 404 returned error can't find the container with id 8a5b0ea75a5a1e7c4e1daf619db65d0b5ea09804d7e617f7ce0767272035c6f6 Jul 08 13:23:47.443272 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:47.443235 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1804049_94ed_4c4c_a122_c8ba0b2ca304.slice/crio-fb900b6120e0d00efa265af582f3af7e032c254541c4b01e529f6cb6ac5e15dc WatchSource:0}: Error finding container fb900b6120e0d00efa265af582f3af7e032c254541c4b01e529f6cb6ac5e15dc: Status 404 returned error can't find the container with id fb900b6120e0d00efa265af582f3af7e032c254541c4b01e529f6cb6ac5e15dc Jul 08 13:23:48.344155 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.344110 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pjblf" event={"ID":"a1804049-94ed-4c4c-a122-c8ba0b2ca304","Type":"ContainerStarted","Data":"fb900b6120e0d00efa265af582f3af7e032c254541c4b01e529f6cb6ac5e15dc"} Jul 08 13:23:48.345601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.345568 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gljg" event={"ID":"c379c49d-db21-4f67-b2a8-b6c871b05a13","Type":"ContainerStarted","Data":"8a5b0ea75a5a1e7c4e1daf619db65d0b5ea09804d7e617f7ce0767272035c6f6"} Jul 08 13:23:48.779424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.779391 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:48.779424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.779432 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:48.782104 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.782078 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"image-registry-7cf9c5fc8d-7drdq\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:48.782241 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.782150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/4312d479-03e5-4b6f-97f3-4a99f77c09b2-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-bhdbq\" (UID: \"4312d479-03e5-4b6f-97f3-4a99f77c09b2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:48.879840 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.879750 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:48.879840 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.879800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:48.882430 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.882399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/43cc3d2a-0702-464a-a055-88824feaaa42-metrics-tls\") pod \"dns-default-4wnf4\" (UID: \"43cc3d2a-0702-464a-a055-88824feaaa42\") " pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:48.882430 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.882413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ac17ea6d-b1f3-498a-a5d1-ee940a61ec70-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-nplcg\" (UID: \"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:48.969151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.969113 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-dhbls\"" Jul 08 13:23:48.976706 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.976669 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:48.991549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.991529 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q5z89\"" Jul 08 13:23:48.999685 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:48.999654 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" Jul 08 13:23:49.037993 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.037408 2576 scope.go:117] "RemoveContainer" containerID="2ec1e6cef259575431ff76e9532116842bf3cc4b19b068841ca86a60c360cbb2" Jul 08 13:23:49.098285 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.098072 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qldzq\"" Jul 08 13:23:49.106016 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.105840 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" Jul 08 13:23:49.126076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.125987 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rlw5\"" Jul 08 13:23:49.134479 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.134410 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4wnf4" Jul 08 13:23:49.141854 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.141769 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:23:49.148311 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:49.148282 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bb0e178_c819_4862_bced_3175df56130e.slice/crio-10470442aac6f9fecd2576d7884911fde2d7952d765e1119740d52253a228a20 WatchSource:0}: Error finding container 10470442aac6f9fecd2576d7884911fde2d7952d765e1119740d52253a228a20: Status 404 returned error can't find the container with id 10470442aac6f9fecd2576d7884911fde2d7952d765e1119740d52253a228a20 Jul 08 13:23:49.158267 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.157334 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq"] Jul 08 13:23:49.281496 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.281379 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-nplcg"] Jul 08 13:23:49.285799 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:49.285765 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac17ea6d_b1f3_498a_a5d1_ee940a61ec70.slice/crio-662e51d0e96f46d47dfb445f3fbf4ad7c3f8ddb93f154d9d175923a3812e5252 WatchSource:0}: Error finding container 662e51d0e96f46d47dfb445f3fbf4ad7c3f8ddb93f154d9d175923a3812e5252: Status 404 returned error can't find the container with id 662e51d0e96f46d47dfb445f3fbf4ad7c3f8ddb93f154d9d175923a3812e5252 Jul 08 13:23:49.311930 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.311897 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4wnf4"] Jul 08 13:23:49.315523 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:49.315486 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cc3d2a_0702_464a_a055_88824feaaa42.slice/crio-88b1c068b77b8bd9b00fe44aa4a664575eb69c52f7c348cc87d5dfc85603decd WatchSource:0}: Error finding container 88b1c068b77b8bd9b00fe44aa4a664575eb69c52f7c348cc87d5dfc85603decd: Status 404 returned error can't find the container with id 88b1c068b77b8bd9b00fe44aa4a664575eb69c52f7c348cc87d5dfc85603decd Jul 08 13:23:49.352511 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.352471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pjblf" event={"ID":"a1804049-94ed-4c4c-a122-c8ba0b2ca304","Type":"ContainerStarted","Data":"5a33057726c0a94cf036cd3ccc0cb17d26d9c3cc4c087d1baeb553cb9172a048"} Jul 08 13:23:49.352958 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.352541 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pjblf" event={"ID":"a1804049-94ed-4c4c-a122-c8ba0b2ca304","Type":"ContainerStarted","Data":"47c5fd1639eb31d666e26108b9a768e1a2f61c53e6d806f88abdcf9221281609"} Jul 08 13:23:49.354033 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.353980 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnf4" event={"ID":"43cc3d2a-0702-464a-a055-88824feaaa42","Type":"ContainerStarted","Data":"88b1c068b77b8bd9b00fe44aa4a664575eb69c52f7c348cc87d5dfc85603decd"} Jul 08 13:23:49.355464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.355403 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" event={"ID":"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70","Type":"ContainerStarted","Data":"662e51d0e96f46d47dfb445f3fbf4ad7c3f8ddb93f154d9d175923a3812e5252"} Jul 08 13:23:49.356917 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.356888 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" event={"ID":"4312d479-03e5-4b6f-97f3-4a99f77c09b2","Type":"ContainerStarted","Data":"49edb3645aa46e20a87e9af153194e71f936b05a5a1c3b2151fe01521c02218b"} Jul 08 13:23:49.358804 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.358783 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" event={"ID":"5bb0e178-c819-4862-bced-3175df56130e","Type":"ContainerStarted","Data":"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda"} Jul 08 13:23:49.358913 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.358827 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:23:49.358913 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.358845 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" event={"ID":"5bb0e178-c819-4862-bced-3175df56130e","Type":"ContainerStarted","Data":"10470442aac6f9fecd2576d7884911fde2d7952d765e1119740d52253a228a20"} Jul 08 13:23:49.361190 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.361155 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:23:49.361286 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.361231 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" event={"ID":"002f971a-3e81-492b-b96e-cc929765f68b","Type":"ContainerStarted","Data":"64e26aa0a8b441c25871e7563f1ab26a539fb964326fec8c67c18908e7a8a8d4"} Jul 08 13:23:49.361556 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.361503 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:49.367439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.367391 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pjblf" podStartSLOduration=66.369845084 podStartE2EDuration="1m7.367373282s" totalImagesPullingTime="997.528198ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:49.366026737 +0000 UTC m=+67.859746570" watchObservedRunningTime="2026-07-08 13:23:49.367373282 +0000 UTC m=+67.861093072" Jul 08 13:23:49.398655 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.398347 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" podStartSLOduration=67.398325911 podStartE2EDuration="1m7.398325911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:49.397523176 +0000 UTC m=+67.891242967" watchObservedRunningTime="2026-07-08 13:23:49.398325911 +0000 UTC m=+67.892045701" Jul 08 13:23:49.422763 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.422680 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" podStartSLOduration=57.480828801 podStartE2EDuration="1m6.422658155s" totalImagesPullingTime="8.941829354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:49.421413855 +0000 UTC m=+67.915133672" watchObservedRunningTime="2026-07-08 13:23:49.422658155 +0000 UTC m=+67.916377946" Jul 08 13:23:49.605068 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.605035 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vkv4v"] Jul 08 13:23:49.608548 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.608464 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.612089 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.612059 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw"] Jul 08 13:23:49.616832 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.616584 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 13:23:49.617941 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.617913 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 13:23:49.618238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.618218 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 13:23:49.618340 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.618267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z4knk\"" Jul 08 13:23:49.618409 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.618391 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 13:23:49.619341 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.619317 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f"] Jul 08 13:23:49.619420 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.619398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.621679 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.621854 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh"] Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.622014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.623072 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.624404 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.624776 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.624920 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.625334 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.625365 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pfpsg\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.625523 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.625537 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-855786878c-jvx86"] Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.627632 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vkv4v"] Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.627709 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.627749 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.627916 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.628056 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.628263 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f"] Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.629026 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 08 13:23:49.630663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.630359 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631037 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw"] Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631053 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631075 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh"] Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631356 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631662 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 13:23:49.632021 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.631702 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 13:23:49.632344 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.632038 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 13:23:49.632344 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.632110 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xfwk6\"" Jul 08 13:23:49.651075 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.651014 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-855786878c-jvx86"] Jul 08 13:23:49.686005 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.685966 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq255\" (UniqueName: \"kubernetes.io/projected/215fdf64-f070-4a64-8697-2cc2f1f49881-kube-api-access-zq255\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.686283 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686264 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686488 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e752497d-7a91-42cc-8130-1ce41265fad7-tmp\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5d7b96-a5a5-469f-99f7-5b5417860995-service-ca-bundle\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686544 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686572 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvvw\" (UniqueName: \"kubernetes.io/projected/bf2289cb-02cc-402e-8aea-6789670c47ee-kube-api-access-spvvw\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686605 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbtw\" (UniqueName: \"kubernetes.io/projected/e752497d-7a91-42cc-8130-1ce41265fad7-kube-api-access-lfbtw\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686635 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/215fdf64-f070-4a64-8697-2cc2f1f49881-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686667 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686692 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-snapshots\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686721 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d17a6d5-89cb-4b70-a32a-584158100116-serving-cert\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck6hs\" (UniqueName: \"kubernetes.io/projected/1d17a6d5-89cb-4b70-a32a-584158100116-kube-api-access-ck6hs\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7gx6\" (UniqueName: \"kubernetes.io/projected/3a5d7b96-a5a5-469f-99f7-5b5417860995-kube-api-access-b7gx6\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686842 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.687603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.686876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-default-certificate\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.688405 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.687395 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-metrics-certs\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.688405 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.687428 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2289cb-02cc-402e-8aea-6789670c47ee-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.688405 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.687521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-stats-auth\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.688405 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.687550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e752497d-7a91-42cc-8130-1ce41265fad7-klusterlet-config\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.688405 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.687576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-tmp\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.708089 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.708052 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv"] Jul 08 13:23:49.711631 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.711603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.719919 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.719525 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 13:23:49.719919 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.719840 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jbp9x\"" Jul 08 13:23:49.720107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.719973 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 13:23:49.720107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.720051 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 13:23:49.720591 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.720422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 13:23:49.729767 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.729737 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv"] Jul 08 13:23:49.788854 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.788819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvvw\" (UniqueName: \"kubernetes.io/projected/bf2289cb-02cc-402e-8aea-6789670c47ee-kube-api-access-spvvw\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.789003 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.788872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbtw\" (UniqueName: \"kubernetes.io/projected/e752497d-7a91-42cc-8130-1ce41265fad7-kube-api-access-lfbtw\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.789003 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.788902 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6sq6\" (UniqueName: \"kubernetes.io/projected/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-kube-api-access-j6sq6\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.789003 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.788934 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/215fdf64-f070-4a64-8697-2cc2f1f49881-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.789142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.789142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789112 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-snapshots\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789147 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d17a6d5-89cb-4b70-a32a-584158100116-serving-cert\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789202 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck6hs\" (UniqueName: \"kubernetes.io/projected/1d17a6d5-89cb-4b70-a32a-584158100116-kube-api-access-ck6hs\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789257 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7gx6\" (UniqueName: \"kubernetes.io/projected/3a5d7b96-a5a5-469f-99f7-5b5417860995-kube-api-access-b7gx6\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789292 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789323 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-default-certificate\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-metrics-certs\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2289cb-02cc-402e-8aea-6789670c47ee-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789388 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-stats-auth\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789413 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e752497d-7a91-42cc-8130-1ce41265fad7-klusterlet-config\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-tmp\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789454 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq255\" (UniqueName: \"kubernetes.io/projected/215fdf64-f070-4a64-8697-2cc2f1f49881-kube-api-access-zq255\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789502 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789539 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.791853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789582 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e752497d-7a91-42cc-8130-1ce41265fad7-tmp\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789610 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5d7b96-a5a5-469f-99f7-5b5417860995-service-ca-bundle\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789639 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789712 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/215fdf64-f070-4a64-8697-2cc2f1f49881-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-snapshots\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.789965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d17a6d5-89cb-4b70-a32a-584158100116-tmp\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.792637 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.790373 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e752497d-7a91-42cc-8130-1ce41265fad7-tmp\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.794150 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.793334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a5d7b96-a5a5-469f-99f7-5b5417860995-service-ca-bundle\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.794150 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.793592 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.794150 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.794107 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d17a6d5-89cb-4b70-a32a-584158100116-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.797032 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.795874 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.799515 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-ca\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.799947 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-metrics-certs\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.800146 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1d17a6d5-89cb-4b70-a32a-584158100116-serving-cert\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.800371 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-default-certificate\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.800757 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e752497d-7a91-42cc-8130-1ce41265fad7-klusterlet-config\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.800941 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.801015 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/3a5d7b96-a5a5-469f-99f7-5b5417860995-stats-auth\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.802205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.801949 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/215fdf64-f070-4a64-8697-2cc2f1f49881-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.806469 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.806441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq255\" (UniqueName: \"kubernetes.io/projected/215fdf64-f070-4a64-8697-2cc2f1f49881-kube-api-access-zq255\") pod \"cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh\" (UID: \"215fdf64-f070-4a64-8697-2cc2f1f49881\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.809165 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.809142 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-v4bwh" Jul 08 13:23:49.810185 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.810126 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvvw\" (UniqueName: \"kubernetes.io/projected/bf2289cb-02cc-402e-8aea-6789670c47ee-kube-api-access-spvvw\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.810484 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.810393 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bf2289cb-02cc-402e-8aea-6789670c47ee-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-ccd956564-cbr2f\" (UID: \"bf2289cb-02cc-402e-8aea-6789670c47ee\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.811135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.811030 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7gx6\" (UniqueName: \"kubernetes.io/projected/3a5d7b96-a5a5-469f-99f7-5b5417860995-kube-api-access-b7gx6\") pod \"router-default-855786878c-jvx86\" (UID: \"3a5d7b96-a5a5-469f-99f7-5b5417860995\") " pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:49.811135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.811043 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck6hs\" (UniqueName: \"kubernetes.io/projected/1d17a6d5-89cb-4b70-a32a-584158100116-kube-api-access-ck6hs\") pod \"insights-operator-5c847dcfc4-vkv4v\" (UID: \"1d17a6d5-89cb-4b70-a32a-584158100116\") " pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.813663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.813638 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbtw\" (UniqueName: \"kubernetes.io/projected/e752497d-7a91-42cc-8130-1ce41265fad7-kube-api-access-lfbtw\") pod \"klusterlet-addon-workmgr-55b6999fc7-8rcnw\" (UID: \"e752497d-7a91-42cc-8130-1ce41265fad7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.890145 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.890105 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6sq6\" (UniqueName: \"kubernetes.io/projected/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-kube-api-access-j6sq6\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.890345 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.890226 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.890345 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.890269 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.891867 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.891812 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.903531 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.902976 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6sq6\" (UniqueName: \"kubernetes.io/projected/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-kube-api-access-j6sq6\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.905332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.905279 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cf4d6258-b250-4459-bc25-e07aa3cbd1f3-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-p8wlv\" (UID: \"cf4d6258-b250-4459-bc25-e07aa3cbd1f3\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:49.930685 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.930432 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" Jul 08 13:23:49.941918 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.941893 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:23:49.961255 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.961225 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" Jul 08 13:23:49.967956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.967930 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" Jul 08 13:23:49.986232 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:49.985461 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:50.011917 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.010495 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-nw47v"] Jul 08 13:23:50.025560 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.024499 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nw47v"] Jul 08 13:23:50.025560 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.024622 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:23:50.025823 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.025576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" Jul 08 13:23:50.029445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.028492 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-prcmd\"" Jul 08 13:23:50.029445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.028848 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 13:23:50.029445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.029269 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 13:23:50.097990 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.097798 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvw6\" (UniqueName: \"kubernetes.io/projected/815297eb-520c-4cc5-aa27-8c53e07d5e27-kube-api-access-nqvw6\") pod \"downloads-685bfb5f64-nw47v\" (UID: \"815297eb-520c-4cc5-aa27-8c53e07d5e27\") " pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:23:50.198978 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.198556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvw6\" (UniqueName: \"kubernetes.io/projected/815297eb-520c-4cc5-aa27-8c53e07d5e27-kube-api-access-nqvw6\") pod \"downloads-685bfb5f64-nw47v\" (UID: \"815297eb-520c-4cc5-aa27-8c53e07d5e27\") " pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:23:50.222082 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.222038 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvw6\" (UniqueName: \"kubernetes.io/projected/815297eb-520c-4cc5-aa27-8c53e07d5e27-kube-api-access-nqvw6\") pod \"downloads-685bfb5f64-nw47v\" (UID: \"815297eb-520c-4cc5-aa27-8c53e07d5e27\") " pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:23:50.247851 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.247244 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f"] Jul 08 13:23:50.252101 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:50.252066 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2289cb_02cc_402e_8aea_6789670c47ee.slice/crio-a57ed78458c00c49d08cc6eeec1133f8bb4acc427bc6a2a0953302e3851a24a2 WatchSource:0}: Error finding container a57ed78458c00c49d08cc6eeec1133f8bb4acc427bc6a2a0953302e3851a24a2: Status 404 returned error can't find the container with id a57ed78458c00c49d08cc6eeec1133f8bb4acc427bc6a2a0953302e3851a24a2 Jul 08 13:23:50.256821 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.256750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw"] Jul 08 13:23:50.292741 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.291639 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh"] Jul 08 13:23:50.300891 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:50.300854 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod215fdf64_f070_4a64_8697_2cc2f1f49881.slice/crio-d74f0e413ed8361bd5100cecc4495cb584a67c8e0e75cb56b6628ef080a17a9f WatchSource:0}: Error finding container d74f0e413ed8361bd5100cecc4495cb584a67c8e0e75cb56b6628ef080a17a9f: Status 404 returned error can't find the container with id d74f0e413ed8361bd5100cecc4495cb584a67c8e0e75cb56b6628ef080a17a9f Jul 08 13:23:50.320136 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.320066 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-855786878c-jvx86"] Jul 08 13:23:50.334963 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:50.334928 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a5d7b96_a5a5_469f_99f7_5b5417860995.slice/crio-189dcf3232abe95eb29b05b5d18a701fc3ad7dae4140ee4ee2d96528d95571fb WatchSource:0}: Error finding container 189dcf3232abe95eb29b05b5d18a701fc3ad7dae4140ee4ee2d96528d95571fb: Status 404 returned error can't find the container with id 189dcf3232abe95eb29b05b5d18a701fc3ad7dae4140ee4ee2d96528d95571fb Jul 08 13:23:50.346055 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.346024 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:23:50.368484 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.367391 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vkv4v"] Jul 08 13:23:50.373019 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.372943 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" event={"ID":"215fdf64-f070-4a64-8697-2cc2f1f49881","Type":"ContainerStarted","Data":"d74f0e413ed8361bd5100cecc4495cb584a67c8e0e75cb56b6628ef080a17a9f"} Jul 08 13:23:50.377812 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.377556 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-855786878c-jvx86" event={"ID":"3a5d7b96-a5a5-469f-99f7-5b5417860995","Type":"ContainerStarted","Data":"189dcf3232abe95eb29b05b5d18a701fc3ad7dae4140ee4ee2d96528d95571fb"} Jul 08 13:23:50.380149 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.379789 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" event={"ID":"bf2289cb-02cc-402e-8aea-6789670c47ee","Type":"ContainerStarted","Data":"a57ed78458c00c49d08cc6eeec1133f8bb4acc427bc6a2a0953302e3851a24a2"} Jul 08 13:23:50.383336 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.383309 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" event={"ID":"e752497d-7a91-42cc-8130-1ce41265fad7","Type":"ContainerStarted","Data":"53174d5f5565572a2d4dc7974272432847518c8c12a8d3b42c420aac13d86ab9"} Jul 08 13:23:50.522641 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.522429 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv"] Jul 08 13:23:50.538213 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:50.538165 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-nw47v"] Jul 08 13:23:50.997252 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:23:50.997152 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815297eb_520c_4cc5_aa27_8c53e07d5e27.slice/crio-aebc49b049888836aea7160f6bd58da3fe9205082ff508f895036d25085b41f3 WatchSource:0}: Error finding container aebc49b049888836aea7160f6bd58da3fe9205082ff508f895036d25085b41f3: Status 404 returned error can't find the container with id aebc49b049888836aea7160f6bd58da3fe9205082ff508f895036d25085b41f3 Jul 08 13:23:51.391998 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.391431 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-855786878c-jvx86" event={"ID":"3a5d7b96-a5a5-469f-99f7-5b5417860995","Type":"ContainerStarted","Data":"5f99dc9fd4903dd52f37c0b9f00b70db6f70237359e2cfc0adb27cdda10705c9"} Jul 08 13:23:51.393087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.393056 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" event={"ID":"cf4d6258-b250-4459-bc25-e07aa3cbd1f3","Type":"ContainerStarted","Data":"0add31afb9fd6f29c62b6869a106e9e1f6449a8af6872d6e580a0adee89dbbb0"} Jul 08 13:23:51.394693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.394631 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" event={"ID":"1d17a6d5-89cb-4b70-a32a-584158100116","Type":"ContainerStarted","Data":"7d315ad7e7dc4ea730e7a9bf04eda5b458c21f364b1ba7b808cdb2d30cf5c59b"} Jul 08 13:23:51.399898 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.399471 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nw47v" event={"ID":"815297eb-520c-4cc5-aa27-8c53e07d5e27","Type":"ContainerStarted","Data":"aebc49b049888836aea7160f6bd58da3fe9205082ff508f895036d25085b41f3"} Jul 08 13:23:51.416915 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.416511 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-855786878c-jvx86" podStartSLOduration=2.416492574 podStartE2EDuration="2.416492574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:23:51.414846096 +0000 UTC m=+69.908565911" watchObservedRunningTime="2026-07-08 13:23:51.416492574 +0000 UTC m=+69.910212344" Jul 08 13:23:51.986453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.986374 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:51.989398 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:51.989359 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:52.405247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:52.405212 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:23:52.407275 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:23:52.407247 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-855786878c-jvx86" Jul 08 13:24:01.461791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.461017 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" event={"ID":"ac17ea6d-b1f3-498a-a5d1-ee940a61ec70","Type":"ContainerStarted","Data":"e6232ef2a72db8884d48a25a893b119a39bbaa63de57c1b1a508f0f82d4c7c30"} Jul 08 13:24:01.472779 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.472748 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" event={"ID":"4312d479-03e5-4b6f-97f3-4a99f77c09b2","Type":"ContainerStarted","Data":"643adf1b0a32b72392497dc5dd3347b61cdcac6fd227c66247437826f9baae3c"} Jul 08 13:24:01.496558 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.495809 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" event={"ID":"bf2289cb-02cc-402e-8aea-6789670c47ee","Type":"ContainerStarted","Data":"cfa53ebf7990194d24a7706667af75c23811288a8db53119b53617a41aea2d6b"} Jul 08 13:24:01.506375 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.503269 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" event={"ID":"e752497d-7a91-42cc-8130-1ce41265fad7","Type":"ContainerStarted","Data":"73210f1c0d2ad394879560891e5445d600fde546d30eebeeaeea28dbf5367aef"} Jul 08 13:24:01.506375 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.504933 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:24:01.509662 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.507939 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" Jul 08 13:24:01.513212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.512368 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-ccd956564-cbr2f" podStartSLOduration=1.575682412 podStartE2EDuration="12.512351044s" totalImagesPullingTime="10.936668632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.511399101 +0000 UTC m=+80.005118891" watchObservedRunningTime="2026-07-08 13:24:01.512351044 +0000 UTC m=+80.006070834" Jul 08 13:24:01.513212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.512730 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" event={"ID":"215fdf64-f070-4a64-8697-2cc2f1f49881","Type":"ContainerStarted","Data":"3a3f2b0e1d54ede3b44d22e6468ad9d561e7f8643bbd4d9263c95951bfe5361c"} Jul 08 13:24:01.513212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.513084 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-nplcg" podStartSLOduration=41.044561487 podStartE2EDuration="46.513075332s" totalImagesPullingTime="5.468513845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.486800764 +0000 UTC m=+79.980520558" watchObservedRunningTime="2026-07-08 13:24:01.513075332 +0000 UTC m=+80.006795122" Jul 08 13:24:01.527684 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.519144 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" event={"ID":"cf4d6258-b250-4459-bc25-e07aa3cbd1f3","Type":"ContainerStarted","Data":"513231de88afde022e3b6539e55db33932518608644757b5d8e70faa7720b19f"} Jul 08 13:24:01.530830 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.530623 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-55b6999fc7-8rcnw" podStartSLOduration=1.578868208 podStartE2EDuration="12.530604604s" totalImagesPullingTime="10.951736396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.529946759 +0000 UTC m=+80.023666563" watchObservedRunningTime="2026-07-08 13:24:01.530604604 +0000 UTC m=+80.024324394" Jul 08 13:24:01.539342 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.539314 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" event={"ID":"1d17a6d5-89cb-4b70-a32a-584158100116","Type":"ContainerStarted","Data":"57c2127a14f3443d43600a220a0a4b8959d95a833e6cf8f327323c5e046e5f0e"} Jul 08 13:24:01.545677 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.545626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8gljg" event={"ID":"c379c49d-db21-4f67-b2a8-b6c871b05a13","Type":"ContainerStarted","Data":"584572561e57a9cd19b43b59800bdb5c4dd806e501143ecf1394a39344ef19a0"} Jul 08 13:24:01.555524 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.555475 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-p8wlv" podStartSLOduration=1.895320119 podStartE2EDuration="12.555437003s" totalImagesPullingTime="10.660116884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.553119286 +0000 UTC m=+80.046839075" watchObservedRunningTime="2026-07-08 13:24:01.555437003 +0000 UTC m=+80.049156794" Jul 08 13:24:01.558746 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.558718 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnf4" event={"ID":"43cc3d2a-0702-464a-a055-88824feaaa42","Type":"ContainerStarted","Data":"7bfbc6beff5c1a2288ac2e001fc6687ed869905a9c29730eb152f838e7a6d3c8"} Jul 08 13:24:01.600217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.597871 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-vkv4v" podStartSLOduration=1.7848607840000001 podStartE2EDuration="12.597846714s" totalImagesPullingTime="10.81298593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.596523539 +0000 UTC m=+80.090243332" watchObservedRunningTime="2026-07-08 13:24:01.597846714 +0000 UTC m=+80.091566504" Jul 08 13:24:01.600217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:01.598043 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8gljg" podStartSLOduration=70.140828621 podStartE2EDuration="1m19.598034162s" totalImagesPullingTime="9.457205541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:01.575028524 +0000 UTC m=+80.068748314" watchObservedRunningTime="2026-07-08 13:24:01.598034162 +0000 UTC m=+80.091753952" Jul 08 13:24:02.230880 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.230839 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jk28s"] Jul 08 13:24:02.233513 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.233487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.236464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.236441 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 13:24:02.236626 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.236514 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvhmk\"" Jul 08 13:24:02.237444 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.237422 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 13:24:02.257697 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.257665 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk28s"] Jul 08 13:24:02.315665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.315622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.315896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.315691 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1353abf6-046c-4345-b792-90b2da7941a7-data-volume\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.315896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.315718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.315896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.315744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wkjw\" (UniqueName: \"kubernetes.io/projected/1353abf6-046c-4345-b792-90b2da7941a7-kube-api-access-9wkjw\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.315896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.315864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1353abf6-046c-4345-b792-90b2da7941a7-crio-socket\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.416594 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.416776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1353abf6-046c-4345-b792-90b2da7941a7-data-volume\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.416776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416648 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.416776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wkjw\" (UniqueName: \"kubernetes.io/projected/1353abf6-046c-4345-b792-90b2da7941a7-kube-api-access-9wkjw\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.416776 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.416742 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 13:24:02.416776 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.416766 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 13:24:02.417031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416803 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1353abf6-046c-4345-b792-90b2da7941a7-crio-socket\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.417031 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.416811 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls podName:1353abf6-046c-4345-b792-90b2da7941a7 nodeName:}" failed. No retries permitted until 2026-07-08 13:24:02.916791665 +0000 UTC m=+81.410511437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk28s" (UID: "1353abf6-046c-4345-b792-90b2da7941a7") : secret "insights-runtime-extractor-tls" not found Jul 08 13:24:02.417031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.416745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1353abf6-046c-4345-b792-90b2da7941a7-crio-socket\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.417031 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.416839 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm podName:1353abf6-046c-4345-b792-90b2da7941a7 nodeName:}" failed. No retries permitted until 2026-07-08 13:24:02.916820641 +0000 UTC m=+81.410540422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk28s" (UID: "1353abf6-046c-4345-b792-90b2da7941a7") : configmap "kube-rbac-proxy" not found Jul 08 13:24:02.417241 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.417074 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1353abf6-046c-4345-b792-90b2da7941a7-data-volume\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.424598 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.424570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wkjw\" (UniqueName: \"kubernetes.io/projected/1353abf6-046c-4345-b792-90b2da7941a7-kube-api-access-9wkjw\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.565831 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.565744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" event={"ID":"215fdf64-f070-4a64-8697-2cc2f1f49881","Type":"ContainerStarted","Data":"2b4750a1f19ae4259e879bf0e6c009df562ea00d3d5fb9d3cb0f4e3bbb39652f"} Jul 08 13:24:02.565831 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.565800 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" event={"ID":"215fdf64-f070-4a64-8697-2cc2f1f49881","Type":"ContainerStarted","Data":"2cac620de5531ab142cdac8be2d8c6df157ec84e5e0b0a99c91f8e4fb1587311"} Jul 08 13:24:02.567989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.567965 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4wnf4" event={"ID":"43cc3d2a-0702-464a-a055-88824feaaa42","Type":"ContainerStarted","Data":"3e3e8a910de78af16c3bd4c182dc01c09533c9007623f34574c58ce6e86896a4"} Jul 08 13:24:02.568129 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.568033 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4wnf4" Jul 08 13:24:02.570269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.570233 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" event={"ID":"4312d479-03e5-4b6f-97f3-4a99f77c09b2","Type":"ContainerStarted","Data":"6ededd9fd372b28345be400218d43026e88189608b129e23964a05c0499e74c9"} Jul 08 13:24:02.585045 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.584989 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7d7c8ff9fb-nrxfh" podStartSLOduration=2.675089269 podStartE2EDuration="13.584969622s" totalImagesPullingTime="10.909880353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:02.583967449 +0000 UTC m=+81.077687240" watchObservedRunningTime="2026-07-08 13:24:02.584969622 +0000 UTC m=+81.078689408" Jul 08 13:24:02.602802 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.602749 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-bhdbq" podStartSLOduration=71.92653862 podStartE2EDuration="1m19.60273395s" totalImagesPullingTime="7.67619533s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:22:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:02.601713513 +0000 UTC m=+81.095433323" watchObservedRunningTime="2026-07-08 13:24:02.60273395 +0000 UTC m=+81.096453747" Jul 08 13:24:02.621606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.621556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4wnf4" podStartSLOduration=36.060908963 podStartE2EDuration="46.621541145s" totalImagesPullingTime="10.560632182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:02.61978303 +0000 UTC m=+81.113502840" watchObservedRunningTime="2026-07-08 13:24:02.621541145 +0000 UTC m=+81.115260934" Jul 08 13:24:02.922077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.922036 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.922291 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:02.922106 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:02.922291 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.922205 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 13:24:02.922291 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.922218 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 13:24:02.922291 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.922269 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm podName:1353abf6-046c-4345-b792-90b2da7941a7 nodeName:}" failed. No retries permitted until 2026-07-08 13:24:03.922252426 +0000 UTC m=+82.415972213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jk28s" (UID: "1353abf6-046c-4345-b792-90b2da7941a7") : configmap "kube-rbac-proxy" not found Jul 08 13:24:02.922291 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:02.922287 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls podName:1353abf6-046c-4345-b792-90b2da7941a7 nodeName:}" failed. No retries permitted until 2026-07-08 13:24:03.922277979 +0000 UTC m=+82.415997746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk28s" (UID: "1353abf6-046c-4345-b792-90b2da7941a7") : secret "insights-runtime-extractor-tls" not found Jul 08 13:24:03.283162 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:03.283081 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnf4_43cc3d2a-0702-464a-a055-88824feaaa42/dns/0.log" Jul 08 13:24:03.483666 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:03.483640 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnf4_43cc3d2a-0702-464a-a055-88824feaaa42/kube-rbac-proxy/0.log" Jul 08 13:24:03.931591 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:03.931544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:03.932052 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:03.931621 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:03.932052 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:03.931687 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 13:24:03.932052 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:03.931788 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls podName:1353abf6-046c-4345-b792-90b2da7941a7 nodeName:}" failed. No retries permitted until 2026-07-08 13:24:05.931766304 +0000 UTC m=+84.425486072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jk28s" (UID: "1353abf6-046c-4345-b792-90b2da7941a7") : secret "insights-runtime-extractor-tls" not found Jul 08 13:24:03.932259 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:03.932064 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1353abf6-046c-4345-b792-90b2da7941a7-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:04.083134 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:04.083101 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cskqh_5058507b-4126-4d96-9618-48ae609b35a3/dns-node-resolver/0.log" Jul 08 13:24:05.086388 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:05.086351 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7cf9c5fc8d-7drdq_5bb0e178-c819-4862-bced-3175df56130e/registry/0.log" Jul 08 13:24:05.683061 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:05.683026 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sgrrx_9508464b-6316-489b-a555-5eb6b3304982/node-ca/0.log" Jul 08 13:24:05.884505 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:05.884476 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855786878c-jvx86_3a5d7b96-a5a5-469f-99f7-5b5417860995/router/0.log" Jul 08 13:24:05.948535 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:05.948451 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:05.950808 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:05.950783 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1353abf6-046c-4345-b792-90b2da7941a7-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jk28s\" (UID: \"1353abf6-046c-4345-b792-90b2da7941a7\") " pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:06.083427 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:06.083350 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rsqrk_6813cb3b-ee4c-405a-b290-7845d985063b/serve-healthcheck-canary/0.log" Jul 08 13:24:06.144581 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:06.144550 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jk28s" Jul 08 13:24:06.276598 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:06.276575 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jk28s"] Jul 08 13:24:06.279125 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:24:06.279096 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1353abf6_046c_4345_b792_90b2da7941a7.slice/crio-b7ca8b7bfd077bf18ff9d27d2c4c83b97ff790c390ba04acf31c2cb6b14117fd WatchSource:0}: Error finding container b7ca8b7bfd077bf18ff9d27d2c4c83b97ff790c390ba04acf31c2cb6b14117fd: Status 404 returned error can't find the container with id b7ca8b7bfd077bf18ff9d27d2c4c83b97ff790c390ba04acf31c2cb6b14117fd Jul 08 13:24:06.588770 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:06.588679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk28s" event={"ID":"1353abf6-046c-4345-b792-90b2da7941a7","Type":"ContainerStarted","Data":"b7ca8b7bfd077bf18ff9d27d2c4c83b97ff790c390ba04acf31c2cb6b14117fd"} Jul 08 13:24:08.982632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:08.982592 2576 patch_prober.go:28] interesting pod/image-registry-7cf9c5fc8d-7drdq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 13:24:08.983115 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:08.982669 2576 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" podUID="5bb0e178-c819-4862-bced-3175df56130e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 13:24:10.389038 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:10.389005 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:24:12.578765 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:12.578729 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4wnf4" Jul 08 13:24:15.344879 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.344530 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5z4pm"] Jul 08 13:24:15.394010 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.393834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.396959 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.396931 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 13:24:15.397500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.397207 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 13:24:15.397500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.397214 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 13:24:15.397500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.397294 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dfn6w\"" Jul 08 13:24:15.397500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.397207 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 13:24:15.541762 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.541647 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-tls\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.541762 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.541715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-metrics-client-ca\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542055 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.541987 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542052 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-sys\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542081 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-textfile\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542117 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-wtmp\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542284 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmthh\" (UniqueName: \"kubernetes.io/projected/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-kube-api-access-qmthh\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.542424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.542337 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-root\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643751 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643714 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643784 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-sys\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643815 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-textfile\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643851 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643900 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-wtmp\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-sys\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.643946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643946 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmthh\" (UniqueName: \"kubernetes.io/projected/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-kube-api-access-qmthh\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.643973 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-root\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644001 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-tls\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644028 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-metrics-client-ca\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644424 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-root\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644440 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-wtmp\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644577 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-metrics-client-ca\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644779 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644761 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-textfile\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.644992 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.644956 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-accelerators-collector-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.647246 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.647221 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-tls\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.647517 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.647488 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.652024 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.652000 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmthh\" (UniqueName: \"kubernetes.io/projected/eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98-kube-api-access-qmthh\") pod \"node-exporter-5z4pm\" (UID: \"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98\") " pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:15.712008 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:15.711974 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5z4pm" Jul 08 13:24:18.130709 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:24:18.130668 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecb7c7c_a1a7_4bd9_9d4c_4b2dddf8fa98.slice/crio-5a6e6906f632a4912ed070984cbf8af29090287f3e8e9ceb539c0099cd8904ab WatchSource:0}: Error finding container 5a6e6906f632a4912ed070984cbf8af29090287f3e8e9ceb539c0099cd8904ab: Status 404 returned error can't find the container with id 5a6e6906f632a4912ed070984cbf8af29090287f3e8e9ceb539c0099cd8904ab Jul 08 13:24:18.348875 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.348843 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njj9z" Jul 08 13:24:18.648912 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.648773 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk28s" event={"ID":"1353abf6-046c-4345-b792-90b2da7941a7","Type":"ContainerStarted","Data":"73f8917d3043178394017e7b9c8073e44e52cf19cbe68f6bf9b5def10a659389"} Jul 08 13:24:18.652438 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.651439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-nw47v" event={"ID":"815297eb-520c-4cc5-aa27-8c53e07d5e27","Type":"ContainerStarted","Data":"11087ba40970b61900efee4a3bea0a8f840c3949c3324c06c7fb19ad9602fcea"} Jul 08 13:24:18.652438 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.651848 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:24:18.654518 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.654488 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z4pm" event={"ID":"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98","Type":"ContainerStarted","Data":"5a6e6906f632a4912ed070984cbf8af29090287f3e8e9ceb539c0099cd8904ab"} Jul 08 13:24:18.666135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.666104 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-nw47v" Jul 08 13:24:18.670202 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:18.670130 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-nw47v" podStartSLOduration=2.500685339 podStartE2EDuration="29.670113326s" totalImagesPullingTime="27.169427987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:18.668294895 +0000 UTC m=+97.162014711" watchObservedRunningTime="2026-07-08 13:24:18.670113326 +0000 UTC m=+97.163833116" Jul 08 13:24:20.666395 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:20.666355 2576 generic.go:356] "Generic (PLEG): container finished" podID="eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98" containerID="d8b6e8fbeea4ce1247a6af4c2897d2974a5a486879f174e49a89f637fe60996d" exitCode=0 Jul 08 13:24:20.666865 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:20.666439 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z4pm" event={"ID":"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98","Type":"ContainerDied","Data":"d8b6e8fbeea4ce1247a6af4c2897d2974a5a486879f174e49a89f637fe60996d"} Jul 08 13:24:20.668813 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:20.668780 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk28s" event={"ID":"1353abf6-046c-4345-b792-90b2da7941a7","Type":"ContainerStarted","Data":"25375c7f03c7620a27633710e79d21120390877327b87c42ad037b87bc64b9c7"} Jul 08 13:24:21.133682 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:21.133459 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:24:21.678640 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:21.678599 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z4pm" event={"ID":"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98","Type":"ContainerStarted","Data":"03ccdc2a0dcb31015770dd1312dd7098c944144a590db5b6897447dc94cfba53"} Jul 08 13:24:21.679101 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:21.678672 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5z4pm" event={"ID":"eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98","Type":"ContainerStarted","Data":"6ea6da3dad73c5a71f4fcddda27ab957d142dc5660e0a9cda99ad335c82ea0d3"} Jul 08 13:24:21.697895 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:21.697829 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5z4pm" podStartSLOduration=4.860406875 podStartE2EDuration="6.697809836s" totalImagesPullingTime="1.710598806s" totalInitContainerRuntime="126.804155ms" isStatefulPod=true podCreationTimestamp="2026-07-08 13:24:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:21.695956942 +0000 UTC m=+100.189676770" watchObservedRunningTime="2026-07-08 13:24:21.697809836 +0000 UTC m=+100.191529625" Jul 08 13:24:24.695925 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:24.695871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jk28s" event={"ID":"1353abf6-046c-4345-b792-90b2da7941a7","Type":"ContainerStarted","Data":"8bebe200522bb09d5db3400c8aae1e945ab9cfcd4606d04cc656d053da9f698d"} Jul 08 13:24:24.713631 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:24.713564 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jk28s" podStartSLOduration=5.523164896 podStartE2EDuration="22.713544641s" totalImagesPullingTime="17.190379745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 13:24:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:24:24.71290366 +0000 UTC m=+103.206623472" watchObservedRunningTime="2026-07-08 13:24:24.713544641 +0000 UTC m=+103.207264430" Jul 08 13:24:32.731740 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:32.731703 2576 generic.go:356] "Generic (PLEG): container finished" podID="a9e657f5-aaf8-4a08-a567-2346b505dfd7" containerID="9e683825d939075549b6554576b771354e39f48b0c94682c4375c91ee7f237b0" exitCode=0 Jul 08 13:24:32.732138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:32.731766 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" event={"ID":"a9e657f5-aaf8-4a08-a567-2346b505dfd7","Type":"ContainerDied","Data":"9e683825d939075549b6554576b771354e39f48b0c94682c4375c91ee7f237b0"} Jul 08 13:24:32.732138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:32.732112 2576 scope.go:117] "RemoveContainer" containerID="9e683825d939075549b6554576b771354e39f48b0c94682c4375c91ee7f237b0" Jul 08 13:24:33.737253 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:33.737219 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-l7dm8" event={"ID":"a9e657f5-aaf8-4a08-a567-2346b505dfd7","Type":"ContainerStarted","Data":"3897b76beb67690391fe17477b42c9e596ada2f6a122eeb39c255abfd610d7ff"} Jul 08 13:24:42.773733 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:42.773695 2576 generic.go:356] "Generic (PLEG): container finished" podID="c624bc17-d20d-4de9-b708-48f183372910" containerID="c7c82c47863df331a320ba82ab9e114027aa7d1d053c3aa9da09d75419b54b9e" exitCode=0 Jul 08 13:24:42.774215 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:42.773743 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" event={"ID":"c624bc17-d20d-4de9-b708-48f183372910","Type":"ContainerDied","Data":"c7c82c47863df331a320ba82ab9e114027aa7d1d053c3aa9da09d75419b54b9e"} Jul 08 13:24:42.774215 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:42.774077 2576 scope.go:117] "RemoveContainer" containerID="c7c82c47863df331a320ba82ab9e114027aa7d1d053c3aa9da09d75419b54b9e" Jul 08 13:24:43.780247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:43.780207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-pnljx" event={"ID":"c624bc17-d20d-4de9-b708-48f183372910","Type":"ContainerStarted","Data":"a5f86afa44915724abfce124e8072dadbf90dc8b8c99c4a6afe0a09eff9c0756"} Jul 08 13:24:46.162288 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.162227 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" podUID="5bb0e178-c819-4862-bced-3175df56130e" containerName="registry" containerID="cri-o://70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda" gracePeriod=30 Jul 08 13:24:46.438946 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.438880 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:24:46.510905 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.510876 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rnwnj\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511063 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.510930 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511063 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.510970 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511063 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511001 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511063 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511041 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511275 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511098 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511275 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511127 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511275 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511157 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets\") pod \"5bb0e178-c819-4862-bced-3175df56130e\" (UID: \"5bb0e178-c819-4862-bced-3175df56130e\") " Jul 08 13:24:46.511588 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511528 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:24:46.511800 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.511571 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:24:46.513641 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.513589 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:24:46.513780 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.513700 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:24:46.513780 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.513757 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:24:46.513872 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.513804 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:24:46.513909 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.513873 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj" (OuterVolumeSpecName: "kube-api-access-rnwnj") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "kube-api-access-rnwnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:24:46.519792 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.519766 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "5bb0e178-c819-4862-bced-3175df56130e" (UID: "5bb0e178-c819-4862-bced-3175df56130e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 13:24:46.612501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612468 2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-image-registry-private-configuration\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612497 2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-bound-sa-token\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612507 2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-trusted-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612514 2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5bb0e178-c819-4862-bced-3175df56130e-registry-certificates\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612523 2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5bb0e178-c819-4862-bced-3175df56130e-ca-trust-extracted\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612531 2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-registry-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612539 2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5bb0e178-c819-4862-bced-3175df56130e-installation-pull-secrets\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.612717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.612549 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnwnj\" (UniqueName: \"kubernetes.io/projected/5bb0e178-c819-4862-bced-3175df56130e-kube-api-access-rnwnj\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:24:46.792383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.792302 2576 generic.go:356] "Generic (PLEG): container finished" podID="5bb0e178-c819-4862-bced-3175df56130e" containerID="70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda" exitCode=0 Jul 08 13:24:46.792383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.792346 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" Jul 08 13:24:46.792383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.792370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" event={"ID":"5bb0e178-c819-4862-bced-3175df56130e","Type":"ContainerDied","Data":"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda"} Jul 08 13:24:46.792586 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.792424 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cf9c5fc8d-7drdq" event={"ID":"5bb0e178-c819-4862-bced-3175df56130e","Type":"ContainerDied","Data":"10470442aac6f9fecd2576d7884911fde2d7952d765e1119740d52253a228a20"} Jul 08 13:24:46.792586 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.792445 2576 scope.go:117] "RemoveContainer" containerID="70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda" Jul 08 13:24:46.801427 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.801410 2576 scope.go:117] "RemoveContainer" containerID="70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda" Jul 08 13:24:46.801678 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:24:46.801655 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda\": container with ID starting with 70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda not found: ID does not exist" containerID="70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda" Jul 08 13:24:46.801745 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.801685 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda"} err="failed to get container status \"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda\": rpc error: code = NotFound desc = could not find container \"70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda\": container with ID starting with 70f62b0d94e7b40c98a4fb6e753a2fd76e1b4c9e5abb2afd2b45271b6cb58fda not found: ID does not exist" Jul 08 13:24:46.813433 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.813412 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:24:46.816853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:46.816834 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cf9c5fc8d-7drdq"] Jul 08 13:24:48.042934 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:24:48.042901 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bb0e178-c819-4862-bced-3175df56130e" path="/var/lib/kubelet/pods/5bb0e178-c819-4862-bced-3175df56130e/volumes" Jul 08 13:27:14.117449 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.117417 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8"] Jul 08 13:27:14.117899 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.117710 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bb0e178-c819-4862-bced-3175df56130e" containerName="registry" Jul 08 13:27:14.117899 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.117720 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bb0e178-c819-4862-bced-3175df56130e" containerName="registry" Jul 08 13:27:14.117899 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.117772 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bb0e178-c819-4862-bced-3175df56130e" containerName="registry" Jul 08 13:27:14.119661 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.119646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.136240 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.136219 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 13:27:14.136432 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.136312 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 13:27:14.136549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.136533 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 13:27:14.136617 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.136534 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 13:27:14.136617 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.136569 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hjkcl\"" Jul 08 13:27:14.157201 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.157156 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8"] Jul 08 13:27:14.289670 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.289638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.289839 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.289686 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-manifests\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.289839 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.289704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-tmp\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.289839 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.289788 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.289839 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.289818 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r57pv\" (UniqueName: \"kubernetes.io/projected/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-kube-api-access-r57pv\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391195 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-manifests\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391366 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391203 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-tmp\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391366 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391366 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r57pv\" (UniqueName: \"kubernetes.io/projected/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-kube-api-access-r57pv\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391523 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.391608 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.391584 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-manifests\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.393639 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.393607 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-tmp\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.393748 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.393678 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-webhook-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.393748 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.393697 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-apiservice-cert\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.400327 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.400301 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r57pv\" (UniqueName: \"kubernetes.io/projected/9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed-kube-api-access-r57pv\") pod \"opendatahub-operator-controller-manager-774b77d499-88ws8\" (UID: \"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed\") " pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.429134 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.429105 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:14.587943 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:14.587904 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8"] Jul 08 13:27:14.590565 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:14.590539 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6e11db_ecc3_4160_bda7_ea2c3f4dd2ed.slice/crio-4f218e8f51b05ccd4ff3f2f9a3628df94a59224ff24ed76d44d606338d554541 WatchSource:0}: Error finding container 4f218e8f51b05ccd4ff3f2f9a3628df94a59224ff24ed76d44d606338d554541: Status 404 returned error can't find the container with id 4f218e8f51b05ccd4ff3f2f9a3628df94a59224ff24ed76d44d606338d554541 Jul 08 13:27:15.364968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:15.364926 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" event={"ID":"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed","Type":"ContainerStarted","Data":"4f218e8f51b05ccd4ff3f2f9a3628df94a59224ff24ed76d44d606338d554541"} Jul 08 13:27:18.378928 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:18.378840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" event={"ID":"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed","Type":"ContainerStarted","Data":"4638474e44c52507af39aa5fb1ebfe522573d497c427c719be60767fccdfa193"} Jul 08 13:27:19.384708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:19.384615 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed" containerID="4638474e44c52507af39aa5fb1ebfe522573d497c427c719be60767fccdfa193" exitCode=0 Jul 08 13:27:19.384708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:19.384699 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" event={"ID":"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed","Type":"ContainerDied","Data":"4638474e44c52507af39aa5fb1ebfe522573d497c427c719be60767fccdfa193"} Jul 08 13:27:20.390995 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:20.390959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" event={"ID":"9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed","Type":"ContainerStarted","Data":"54f5957124ec6e96842edd65bb22651142c30c5f90eb6d1b3666b6c527da63d5"} Jul 08 13:27:20.391398 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:20.391085 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:20.413347 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:20.413297 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" podStartSLOduration=1.712723363 podStartE2EDuration="6.41328303s" totalImagesPullingTime="3.695569986s" totalInitContainerRuntime="1.004989681s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:20.411707287 +0000 UTC m=+278.905427087" watchObservedRunningTime="2026-07-08 13:27:20.41328303 +0000 UTC m=+278.907002819" Jul 08 13:27:31.397497 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:31.397465 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-774b77d499-88ws8" Jul 08 13:27:33.987997 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:33.987958 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746f76655b-jzrxr"] Jul 08 13:27:33.997554 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:33.997531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.000505 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.000482 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 08 13:27:34.000890 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.000873 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 13:27:34.003321 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.003301 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f76655b-jzrxr"] Jul 08 13:27:34.037945 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.037910 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6bvq\" (UniqueName: \"kubernetes.io/projected/1a392e64-8b97-4d85-8327-0d784846b4f7-kube-api-access-w6bvq\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.038072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.038004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a392e64-8b97-4d85-8327-0d784846b4f7-tls-certs\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.038072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.038038 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a392e64-8b97-4d85-8327-0d784846b4f7-tmp\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.139413 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.139381 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6bvq\" (UniqueName: \"kubernetes.io/projected/1a392e64-8b97-4d85-8327-0d784846b4f7-kube-api-access-w6bvq\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.139589 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.139455 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a392e64-8b97-4d85-8327-0d784846b4f7-tls-certs\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.139589 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.139477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a392e64-8b97-4d85-8327-0d784846b4f7-tmp\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.141650 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.141622 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1a392e64-8b97-4d85-8327-0d784846b4f7-tmp\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.141955 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.141933 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a392e64-8b97-4d85-8327-0d784846b4f7-tls-certs\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.149651 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.149623 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6bvq\" (UniqueName: \"kubernetes.io/projected/1a392e64-8b97-4d85-8327-0d784846b4f7-kube-api-access-w6bvq\") pod \"kube-auth-proxy-746f76655b-jzrxr\" (UID: \"1a392e64-8b97-4d85-8327-0d784846b4f7\") " pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.308453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.308369 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" Jul 08 13:27:34.356790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.356758 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntkrv"] Jul 08 13:27:34.359467 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.359446 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.365720 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.365697 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h4rms\"" Jul 08 13:27:34.365952 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.365937 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 13:27:34.376939 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.376907 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntkrv"] Jul 08 13:27:34.442230 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.442199 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e35d827-31dc-4c56-bc08-b865ca5337e3-cert\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.442397 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.442304 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzzml\" (UniqueName: \"kubernetes.io/projected/8e35d827-31dc-4c56-bc08-b865ca5337e3-kube-api-access-rzzml\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.444689 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.444670 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746f76655b-jzrxr"] Jul 08 13:27:34.447554 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:34.447525 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a392e64_8b97_4d85_8327_0d784846b4f7.slice/crio-7363d0ceff27a063bceef647f31253264e28c1e2e3a4205933ed48c8328b3dee WatchSource:0}: Error finding container 7363d0ceff27a063bceef647f31253264e28c1e2e3a4205933ed48c8328b3dee: Status 404 returned error can't find the container with id 7363d0ceff27a063bceef647f31253264e28c1e2e3a4205933ed48c8328b3dee Jul 08 13:27:34.543656 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.543624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e35d827-31dc-4c56-bc08-b865ca5337e3-cert\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.543808 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.543679 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzzml\" (UniqueName: \"kubernetes.io/projected/8e35d827-31dc-4c56-bc08-b865ca5337e3-kube-api-access-rzzml\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.546006 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.545988 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e35d827-31dc-4c56-bc08-b865ca5337e3-cert\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.556155 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.556131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzzml\" (UniqueName: \"kubernetes.io/projected/8e35d827-31dc-4c56-bc08-b865ca5337e3-kube-api-access-rzzml\") pod \"odh-model-controller-7998c67dc4-ntkrv\" (UID: \"8e35d827-31dc-4c56-bc08-b865ca5337e3\") " pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.673827 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.673796 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:34.807838 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:34.807729 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-ntkrv"] Jul 08 13:27:34.810623 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:34.810583 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e35d827_31dc_4c56_bc08_b865ca5337e3.slice/crio-5c88b10313b6f5163f4bf246d9b86fd511eac65cbe26373206b2dccd0be0a77a WatchSource:0}: Error finding container 5c88b10313b6f5163f4bf246d9b86fd511eac65cbe26373206b2dccd0be0a77a: Status 404 returned error can't find the container with id 5c88b10313b6f5163f4bf246d9b86fd511eac65cbe26373206b2dccd0be0a77a Jul 08 13:27:35.118067 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.117993 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qzhc4"] Jul 08 13:27:35.124323 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.123312 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.127040 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.127015 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 13:27:35.133037 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.133016 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qzhc4"] Jul 08 13:27:35.148402 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.148373 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/8943a185-42de-4cb3-bff5-d43733e20c47-kube-api-access-bc9jw\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.148559 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.148421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.249306 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.249270 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/8943a185-42de-4cb3-bff5-d43733e20c47-kube-api-access-bc9jw\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.249489 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.249314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.249489 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:27:35.249409 2576 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 08 13:27:35.249489 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:27:35.249467 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs podName:8943a185-42de-4cb3-bff5-d43733e20c47 nodeName:}" failed. No retries permitted until 2026-07-08 13:27:35.749450348 +0000 UTC m=+294.243170115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs") pod "model-serving-api-54d89bd5fc-qzhc4" (UID: "8943a185-42de-4cb3-bff5-d43733e20c47") : secret "model-serving-api-tls" not found Jul 08 13:27:35.268491 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.268339 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc9jw\" (UniqueName: \"kubernetes.io/projected/8943a185-42de-4cb3-bff5-d43733e20c47-kube-api-access-bc9jw\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.356417 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.356381 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:35.360242 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.360216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.362996 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.362968 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 13:27:35.363215 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.363195 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 08 13:27:35.363445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.363427 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 08 13:27:35.363499 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.363458 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 08 13:27:35.363631 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.363611 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bx88f\"" Jul 08 13:27:35.376370 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.376343 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:35.450632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450598 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450815 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450646 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450815 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450681 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450815 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmntb\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450964 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450964 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450839 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.450964 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.450864 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.455371 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.455340 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" event={"ID":"1a392e64-8b97-4d85-8327-0d784846b4f7","Type":"ContainerStarted","Data":"7363d0ceff27a063bceef647f31253264e28c1e2e3a4205933ed48c8328b3dee"} Jul 08 13:27:35.456717 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.456694 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" event={"ID":"8e35d827-31dc-4c56-bc08-b865ca5337e3","Type":"ContainerStarted","Data":"5c88b10313b6f5163f4bf246d9b86fd511eac65cbe26373206b2dccd0be0a77a"} Jul 08 13:27:35.552191 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552049 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552191 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552191 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552139 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmntb\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552251 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552273 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.552826 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.552799 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.555654 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.555608 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.556004 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.555960 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.556620 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.556597 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.556943 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.556921 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.562266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.562239 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.562895 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.562870 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmntb\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb\") pod \"istiod-openshift-gateway-7574459656-fg5jt\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.674214 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.674148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:35.753949 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.753922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.756415 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.756367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8943a185-42de-4cb3-bff5-d43733e20c47-tls-certs\") pod \"model-serving-api-54d89bd5fc-qzhc4\" (UID: \"8943a185-42de-4cb3-bff5-d43733e20c47\") " pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:35.823932 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:35.823906 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:35.826229 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:35.826200 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33da9397_015d_4370_a807_dda10a201e88.slice/crio-e333647838ddfeed5361f54044f2eb0c5827b803f500b36815f8a4ac2a007662 WatchSource:0}: Error finding container e333647838ddfeed5361f54044f2eb0c5827b803f500b36815f8a4ac2a007662: Status 404 returned error can't find the container with id e333647838ddfeed5361f54044f2eb0c5827b803f500b36815f8a4ac2a007662 Jul 08 13:27:36.040430 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:36.040273 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:36.210368 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:36.210280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qzhc4"] Jul 08 13:27:36.214642 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:36.214384 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8943a185_42de_4cb3_bff5_d43733e20c47.slice/crio-615ef044fbe886c8cf71f8fb91e929600ea77c76bb60628b6ae4b91dbd41690e WatchSource:0}: Error finding container 615ef044fbe886c8cf71f8fb91e929600ea77c76bb60628b6ae4b91dbd41690e: Status 404 returned error can't find the container with id 615ef044fbe886c8cf71f8fb91e929600ea77c76bb60628b6ae4b91dbd41690e Jul 08 13:27:36.463529 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:36.463479 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" event={"ID":"8943a185-42de-4cb3-bff5-d43733e20c47","Type":"ContainerStarted","Data":"615ef044fbe886c8cf71f8fb91e929600ea77c76bb60628b6ae4b91dbd41690e"} Jul 08 13:27:36.465050 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:36.464998 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" event={"ID":"33da9397-015d-4370-a807-dda10a201e88","Type":"ContainerStarted","Data":"e333647838ddfeed5361f54044f2eb0c5827b803f500b36815f8a4ac2a007662"} Jul 08 13:27:40.307962 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.307923 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:27:40.308266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.307993 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:27:40.487241 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.487202 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" event={"ID":"33da9397-015d-4370-a807-dda10a201e88","Type":"ContainerStarted","Data":"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655"} Jul 08 13:27:40.487410 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.487349 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:40.489151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.489120 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" event={"ID":"8e35d827-31dc-4c56-bc08-b865ca5337e3","Type":"ContainerStarted","Data":"e854ea5597f687849f3e594b3086b50d0a2fee6bc3a1fb4178e7c2491d7bdf94"} Jul 08 13:27:40.489304 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.489191 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:40.490844 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.490816 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" event={"ID":"1a392e64-8b97-4d85-8327-0d784846b4f7","Type":"ContainerStarted","Data":"3186c798a1ea17ea6448b49506af92ebaf243171005d83da5cf7b573a0c95980"} Jul 08 13:27:40.492546 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.492521 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" event={"ID":"8943a185-42de-4cb3-bff5-d43733e20c47","Type":"ContainerStarted","Data":"93dee8dd40aa9286024bdafb1e26d1d68edc9a8555fc790278a898cd81e5d6bc"} Jul 08 13:27:40.492739 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.492724 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:27:40.509958 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.509912 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" podStartSLOduration=1.030401869 podStartE2EDuration="5.509897347s" totalImagesPullingTime="4.479495478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:40.508484712 +0000 UTC m=+299.002204484" watchObservedRunningTime="2026-07-08 13:27:40.509897347 +0000 UTC m=+299.003617136" Jul 08 13:27:40.527112 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.527062 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" podStartSLOduration=1.43617002 podStartE2EDuration="5.527048388s" totalImagesPullingTime="4.090878368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:40.526062335 +0000 UTC m=+299.019782149" watchObservedRunningTime="2026-07-08 13:27:40.527048388 +0000 UTC m=+299.020768177" Jul 08 13:27:40.544499 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.544398 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746f76655b-jzrxr" podStartSLOduration=1.686921583 podStartE2EDuration="7.544378825s" totalImagesPullingTime="5.857457242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:40.542463908 +0000 UTC m=+299.036183730" watchObservedRunningTime="2026-07-08 13:27:40.544378825 +0000 UTC m=+299.038098619" Jul 08 13:27:40.562955 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:40.562907 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" podStartSLOduration=1.056444153 podStartE2EDuration="6.562893738s" totalImagesPullingTime="5.506449585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:40.561719578 +0000 UTC m=+299.055439368" watchObservedRunningTime="2026-07-08 13:27:40.562893738 +0000 UTC m=+299.056613527" Jul 08 13:27:41.498924 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:41.498891 2576 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-fg5jt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 13:27:41.499325 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:41.498943 2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" podUID="33da9397-015d-4370-a807-dda10a201e88" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 13:27:41.935668 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:41.935638 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:27:41.941798 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:41.941767 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:27:41.946624 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:41.946606 2576 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 13:27:44.497752 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.497721 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:44.864553 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.864469 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv"] Jul 08 13:27:44.867854 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.867834 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:44.870518 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.870496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lc5hv\"" Jul 08 13:27:44.870763 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.870747 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 13:27:44.883510 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.883484 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv"] Jul 08 13:27:44.939569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.939529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g6j9\" (UniqueName: \"kubernetes.io/projected/e22ac27f-8a4b-4643-bb62-89bb2366dadc-kube-api-access-8g6j9\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:44.939749 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:44.939656 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.040590 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.040556 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.040798 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.040629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g6j9\" (UniqueName: \"kubernetes.io/projected/e22ac27f-8a4b-4643-bb62-89bb2366dadc-kube-api-access-8g6j9\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.040798 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:27:45.040716 2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 08 13:27:45.040798 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:27:45.040782 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert podName:e22ac27f-8a4b-4643-bb62-89bb2366dadc nodeName:}" failed. No retries permitted until 2026-07-08 13:27:45.540762383 +0000 UTC m=+304.034482168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert") pod "kserve-controller-manager-d6f4b65b8-cf2hv" (UID: "e22ac27f-8a4b-4643-bb62-89bb2366dadc") : secret "kserve-webhook-server-cert" not found Jul 08 13:27:45.058881 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.058854 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g6j9\" (UniqueName: \"kubernetes.io/projected/e22ac27f-8a4b-4643-bb62-89bb2366dadc-kube-api-access-8g6j9\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.545047 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.545012 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.547413 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.547389 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e22ac27f-8a4b-4643-bb62-89bb2366dadc-cert\") pod \"kserve-controller-manager-d6f4b65b8-cf2hv\" (UID: \"e22ac27f-8a4b-4643-bb62-89bb2366dadc\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.778794 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.778744 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:45.911234 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.911198 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv"] Jul 08 13:27:45.914166 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:27:45.914134 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22ac27f_8a4b_4643_bb62_89bb2366dadc.slice/crio-e66021db47544e9056d30fb1670dd10bbf66f4147360720f9933dc43b849b146 WatchSource:0}: Error finding container e66021db47544e9056d30fb1670dd10bbf66f4147360720f9933dc43b849b146: Status 404 returned error can't find the container with id e66021db47544e9056d30fb1670dd10bbf66f4147360720f9933dc43b849b146 Jul 08 13:27:45.915815 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:45.915795 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:27:46.519847 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:46.519802 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" event={"ID":"e22ac27f-8a4b-4643-bb62-89bb2366dadc","Type":"ContainerStarted","Data":"e66021db47544e9056d30fb1670dd10bbf66f4147360720f9933dc43b849b146"} Jul 08 13:27:47.084931 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.084901 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:47.085366 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.085075 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" podUID="33da9397-015d-4370-a807-dda10a201e88" containerName="discovery" containerID="cri-o://8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655" gracePeriod=30 Jul 08 13:27:47.336061 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.336006 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:47.361255 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361222 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361426 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361271 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmntb\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361426 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361312 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361426 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361382 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361426 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.361606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.361472 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token\") pod \"33da9397-015d-4370-a807-dda10a201e88\" (UID: \"33da9397-015d-4370-a807-dda10a201e88\") " Jul 08 13:27:47.362621 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.362547 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:27:47.364132 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.364090 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs" (OuterVolumeSpecName: "local-certs") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 13:27:47.364132 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.364110 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts" (OuterVolumeSpecName: "cacerts") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:27:47.364132 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.364250 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:27:47.364551 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.364388 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:27:47.364551 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.364490 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token" (OuterVolumeSpecName: "istio-token") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:27:47.365716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.365694 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb" (OuterVolumeSpecName: "kube-api-access-lmntb") pod "33da9397-015d-4370-a807-dda10a201e88" (UID: "33da9397-015d-4370-a807-dda10a201e88"). InnerVolumeSpecName "kube-api-access-lmntb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:27:47.462461 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462423 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmntb\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-kube-api-access-lmntb\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462461 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462454 2576 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/33da9397-015d-4370-a807-dda10a201e88-local-certs\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462461 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462463 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/33da9397-015d-4370-a807-dda10a201e88-istio-csr-ca-configmap\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462461 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462472 2576 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-kubeconfig\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462481 2576 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-istio-csr-dns-cert\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462492 2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33da9397-015d-4370-a807-dda10a201e88-istio-token\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.462716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.462499 2576 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/33da9397-015d-4370-a807-dda10a201e88-cacerts\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:27:47.532995 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.532956 2576 generic.go:356] "Generic (PLEG): container finished" podID="33da9397-015d-4370-a807-dda10a201e88" containerID="8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655" exitCode=0 Jul 08 13:27:47.533194 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.533001 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" event={"ID":"33da9397-015d-4370-a807-dda10a201e88","Type":"ContainerDied","Data":"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655"} Jul 08 13:27:47.533194 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.533031 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" Jul 08 13:27:47.533194 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.533083 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt" event={"ID":"33da9397-015d-4370-a807-dda10a201e88","Type":"ContainerDied","Data":"e333647838ddfeed5361f54044f2eb0c5827b803f500b36815f8a4ac2a007662"} Jul 08 13:27:47.533194 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.533110 2576 scope.go:117] "RemoveContainer" containerID="8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655" Jul 08 13:27:47.543638 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.543611 2576 scope.go:117] "RemoveContainer" containerID="8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655" Jul 08 13:27:47.544003 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:27:47.543974 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655\": container with ID starting with 8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655 not found: ID does not exist" containerID="8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655" Jul 08 13:27:47.544114 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.544013 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655"} err="failed to get container status \"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655\": rpc error: code = NotFound desc = could not find container \"8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655\": container with ID starting with 8d435045708e204d9bda83f80dec2c946782de8842395f2f857b981231fde655 not found: ID does not exist" Jul 08 13:27:47.558035 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.558005 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:47.562445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:47.562423 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-fg5jt"] Jul 08 13:27:48.046829 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:48.046796 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33da9397-015d-4370-a807-dda10a201e88" path="/var/lib/kubelet/pods/33da9397-015d-4370-a807-dda10a201e88/volumes" Jul 08 13:27:49.544769 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:49.544736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" event={"ID":"e22ac27f-8a4b-4643-bb62-89bb2366dadc","Type":"ContainerStarted","Data":"a926fd8d87043091f5bb085c514bcbd11655e3e9ea65e8cc91e8f7f2c44946c9"} Jul 08 13:27:49.545112 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:49.544857 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:27:49.562329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:49.562282 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" podStartSLOduration=2.498277038 podStartE2EDuration="5.562267843s" totalImagesPullingTime="3.063990805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:27:49.560925742 +0000 UTC m=+308.054645554" watchObservedRunningTime="2026-07-08 13:27:49.562267843 +0000 UTC m=+308.055987632" Jul 08 13:27:51.499292 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:51.499256 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-ntkrv" Jul 08 13:27:51.502280 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:27:51.502257 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qzhc4" Jul 08 13:28:20.554507 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:20.554474 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cf2hv" Jul 08 13:28:35.364662 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.364582 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:35.365127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.364956 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33da9397-015d-4370-a807-dda10a201e88" containerName="discovery" Jul 08 13:28:35.365127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.364968 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="33da9397-015d-4370-a807-dda10a201e88" containerName="discovery" Jul 08 13:28:35.365127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.365043 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="33da9397-015d-4370-a807-dda10a201e88" containerName="discovery" Jul 08 13:28:35.367831 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.367810 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:35.370884 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.370858 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 13:28:35.378102 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.378078 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zxlzq\"" Jul 08 13:28:35.378448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.378431 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 13:28:35.392934 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.392908 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:35.456085 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.456053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcm6t\" (UniqueName: \"kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t\") pod \"kuadrant-operator-catalog-52bn9\" (UID: \"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14\") " pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:35.557018 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.556986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcm6t\" (UniqueName: \"kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t\") pod \"kuadrant-operator-catalog-52bn9\" (UID: \"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14\") " pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:35.565903 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.565877 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcm6t\" (UniqueName: \"kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t\") pod \"kuadrant-operator-catalog-52bn9\" (UID: \"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14\") " pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:35.676645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.676619 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:35.709663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.709631 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:35.805354 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.805332 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:35.807645 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:28:35.807617 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda327ebbf_dd5d_45a8_bd8f_1e2f5485ef14.slice/crio-8bcb3f505a3966d7f064330f6b54202439462574fb050fbb6802443f1315cd44 WatchSource:0}: Error finding container 8bcb3f505a3966d7f064330f6b54202439462574fb050fbb6802443f1315cd44: Status 404 returned error can't find the container with id 8bcb3f505a3966d7f064330f6b54202439462574fb050fbb6802443f1315cd44 Jul 08 13:28:35.916802 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.916769 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zxm65"] Jul 08 13:28:35.920960 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.920942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:35.926926 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.926855 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zxm65"] Jul 08 13:28:35.960709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:35.960679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc8n6\" (UniqueName: \"kubernetes.io/projected/4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace-kube-api-access-qc8n6\") pod \"kuadrant-operator-catalog-zxm65\" (UID: \"4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace\") " pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:36.061813 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.061777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc8n6\" (UniqueName: \"kubernetes.io/projected/4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace-kube-api-access-qc8n6\") pod \"kuadrant-operator-catalog-zxm65\" (UID: \"4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace\") " pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:36.069860 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.069828 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc8n6\" (UniqueName: \"kubernetes.io/projected/4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace-kube-api-access-qc8n6\") pod \"kuadrant-operator-catalog-zxm65\" (UID: \"4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace\") " pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:36.232049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.231968 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:36.371154 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.371132 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zxm65"] Jul 08 13:28:36.373387 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:28:36.373362 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f79b658_c2ba_4d2e_8a47_9ef5a3e4dace.slice/crio-3e91fe4fbc93b40be6dea138e636cdb04b55133f436ee434865bf0fd6ab15ce9 WatchSource:0}: Error finding container 3e91fe4fbc93b40be6dea138e636cdb04b55133f436ee434865bf0fd6ab15ce9: Status 404 returned error can't find the container with id 3e91fe4fbc93b40be6dea138e636cdb04b55133f436ee434865bf0fd6ab15ce9 Jul 08 13:28:36.751114 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.751078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" event={"ID":"4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace","Type":"ContainerStarted","Data":"3e91fe4fbc93b40be6dea138e636cdb04b55133f436ee434865bf0fd6ab15ce9"} Jul 08 13:28:36.752433 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:36.752401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" event={"ID":"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14","Type":"ContainerStarted","Data":"8bcb3f505a3966d7f064330f6b54202439462574fb050fbb6802443f1315cd44"} Jul 08 13:28:38.764873 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:38.764832 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" event={"ID":"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14","Type":"ContainerStarted","Data":"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece"} Jul 08 13:28:38.765375 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:38.764852 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" podUID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" containerName="registry-server" containerID="cri-o://050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece" gracePeriod=2 Jul 08 13:28:38.766374 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:38.766349 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" event={"ID":"4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace","Type":"ContainerStarted","Data":"ea2b4212e2cc4ccf4765d098902d74cc6f41d17aab1f2c9c2829206402b77e81"} Jul 08 13:28:38.780821 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:38.780778 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" podStartSLOduration=1.39540971 podStartE2EDuration="3.780761982s" totalImagesPullingTime="2.385352272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:28:38.779691652 +0000 UTC m=+357.273411469" watchObservedRunningTime="2026-07-08 13:28:38.780761982 +0000 UTC m=+357.274481771" Jul 08 13:28:38.794008 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:38.793963 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" podStartSLOduration=1.971509394 podStartE2EDuration="3.7939484s" totalImagesPullingTime="1.822439006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:28:38.793730849 +0000 UTC m=+357.287450639" watchObservedRunningTime="2026-07-08 13:28:38.7939484 +0000 UTC m=+357.287668190" Jul 08 13:28:39.026418 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.026360 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:39.084907 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.084877 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcm6t\" (UniqueName: \"kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t\") pod \"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14\" (UID: \"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14\") " Jul 08 13:28:39.087092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.087066 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t" (OuterVolumeSpecName: "kube-api-access-kcm6t") pod "a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" (UID: "a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14"). InnerVolumeSpecName "kube-api-access-kcm6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:28:39.188206 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.187052 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcm6t\" (UniqueName: \"kubernetes.io/projected/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14-kube-api-access-kcm6t\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:28:39.771605 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.771571 2576 generic.go:356] "Generic (PLEG): container finished" podID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" containerID="050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece" exitCode=0 Jul 08 13:28:39.772036 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.771623 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" Jul 08 13:28:39.772036 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.771652 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" event={"ID":"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14","Type":"ContainerDied","Data":"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece"} Jul 08 13:28:39.772036 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.771723 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-52bn9" event={"ID":"a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14","Type":"ContainerDied","Data":"8bcb3f505a3966d7f064330f6b54202439462574fb050fbb6802443f1315cd44"} Jul 08 13:28:39.772036 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.771744 2576 scope.go:117] "RemoveContainer" containerID="050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece" Jul 08 13:28:39.780938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.780922 2576 scope.go:117] "RemoveContainer" containerID="050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece" Jul 08 13:28:39.781217 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:28:39.781200 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece\": container with ID starting with 050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece not found: ID does not exist" containerID="050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece" Jul 08 13:28:39.781270 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.781226 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece"} err="failed to get container status \"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece\": rpc error: code = NotFound desc = could not find container \"050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece\": container with ID starting with 050f2568a4a5289db51baafa923c649c228a8386de1f9f3391dec6133a19cece not found: ID does not exist" Jul 08 13:28:39.792389 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.792366 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:39.794464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:39.794444 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-52bn9"] Jul 08 13:28:40.042561 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:40.042473 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" path="/var/lib/kubelet/pods/a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14/volumes" Jul 08 13:28:46.232846 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:46.232811 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:46.233362 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:46.232876 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:46.255186 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:46.255147 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:28:46.824459 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:28:46.824430 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zxm65" Jul 08 13:29:07.935252 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.935202 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6"] Jul 08 13:29:07.935708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.935546 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" containerName="registry-server" Jul 08 13:29:07.935708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.935557 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" containerName="registry-server" Jul 08 13:29:07.935708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.935626 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a327ebbf-dd5d-45a8-bd8f-1e2f5485ef14" containerName="registry-server" Jul 08 13:29:07.938928 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.938911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:07.941436 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.941414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l8qx6\"" Jul 08 13:29:07.946823 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:07.946803 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6"] Jul 08 13:29:08.033493 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.033460 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grglv\" (UniqueName: \"kubernetes.io/projected/ba2186af-8af3-45d1-a5f0-2143be5b54ae-kube-api-access-grglv\") pod \"authorino-operator-75cc8d5cb7-tzvx6\" (UID: \"ba2186af-8af3-45d1-a5f0-2143be5b54ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:08.133934 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.133899 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grglv\" (UniqueName: \"kubernetes.io/projected/ba2186af-8af3-45d1-a5f0-2143be5b54ae-kube-api-access-grglv\") pod \"authorino-operator-75cc8d5cb7-tzvx6\" (UID: \"ba2186af-8af3-45d1-a5f0-2143be5b54ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:08.144181 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.144144 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grglv\" (UniqueName: \"kubernetes.io/projected/ba2186af-8af3-45d1-a5f0-2143be5b54ae-kube-api-access-grglv\") pod \"authorino-operator-75cc8d5cb7-tzvx6\" (UID: \"ba2186af-8af3-45d1-a5f0-2143be5b54ae\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:08.249608 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.249502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:08.385025 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.385000 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6"] Jul 08 13:29:08.387787 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:08.387752 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2186af_8af3_45d1_a5f0_2143be5b54ae.slice/crio-cd5fd44e8f0403fe927bf5053ca056c19b6df1d8c87151e556a6e4f5c23664e7 WatchSource:0}: Error finding container cd5fd44e8f0403fe927bf5053ca056c19b6df1d8c87151e556a6e4f5c23664e7: Status 404 returned error can't find the container with id cd5fd44e8f0403fe927bf5053ca056c19b6df1d8c87151e556a6e4f5c23664e7 Jul 08 13:29:08.906704 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:08.906668 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" event={"ID":"ba2186af-8af3-45d1-a5f0-2143be5b54ae","Type":"ContainerStarted","Data":"cd5fd44e8f0403fe927bf5053ca056c19b6df1d8c87151e556a6e4f5c23664e7"} Jul 08 13:29:10.917193 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:10.917131 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" event={"ID":"ba2186af-8af3-45d1-a5f0-2143be5b54ae","Type":"ContainerStarted","Data":"612a0ba3f7536ff8c4789d5ee951e9c7ea68a7effdc2bc0d9981e0f9a1f743bc"} Jul 08 13:29:10.917569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:10.917226 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:10.935039 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:10.934983 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" podStartSLOduration=1.697093545 podStartE2EDuration="3.934967077s" totalImagesPullingTime="2.237873532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:10.933166493 +0000 UTC m=+389.426886281" watchObservedRunningTime="2026-07-08 13:29:10.934967077 +0000 UTC m=+389.428686866" Jul 08 13:29:12.997465 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:12.997431 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz"] Jul 08 13:29:13.001305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.001287 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.006522 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.006497 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-h4s79\"" Jul 08 13:29:13.017969 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.017944 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz"] Jul 08 13:29:13.175918 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.175876 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dllhg\" (UniqueName: \"kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.176124 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.175964 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.277379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.277287 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dllhg\" (UniqueName: \"kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.277379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.277376 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.277762 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.277741 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.289864 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.289835 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dllhg\" (UniqueName: \"kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg\") pod \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.310650 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.310623 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:13.443967 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.443858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz"] Jul 08 13:29:13.446855 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:13.446832 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6dbadb_425e_4364_b430_172259867091.slice/crio-b6370117403ef5f742fc8cd2d3c58e9f1a7169e925f4cbd4620b2f8519b5a3e4 WatchSource:0}: Error finding container b6370117403ef5f742fc8cd2d3c58e9f1a7169e925f4cbd4620b2f8519b5a3e4: Status 404 returned error can't find the container with id b6370117403ef5f742fc8cd2d3c58e9f1a7169e925f4cbd4620b2f8519b5a3e4 Jul 08 13:29:13.931661 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:13.931624 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" event={"ID":"0f6dbadb-425e-4364-b430-172259867091","Type":"ContainerStarted","Data":"b6370117403ef5f742fc8cd2d3c58e9f1a7169e925f4cbd4620b2f8519b5a3e4"} Jul 08 13:29:15.859303 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:15.859264 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw"] Jul 08 13:29:15.862312 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:15.862284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:15.864776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:15.864741 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 13:29:15.864911 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:15.864857 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q9lvl\"" Jul 08 13:29:15.872490 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:15.872339 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw"] Jul 08 13:29:16.002678 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.002642 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6fk\" (UniqueName: \"kubernetes.io/projected/8a73aae9-4d19-4b1f-b797-caa2488003d2-kube-api-access-tb6fk\") pod \"dns-operator-controller-manager-864dd78c4c-55wfw\" (UID: \"8a73aae9-4d19-4b1f-b797-caa2488003d2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:16.103217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.103165 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6fk\" (UniqueName: \"kubernetes.io/projected/8a73aae9-4d19-4b1f-b797-caa2488003d2-kube-api-access-tb6fk\") pod \"dns-operator-controller-manager-864dd78c4c-55wfw\" (UID: \"8a73aae9-4d19-4b1f-b797-caa2488003d2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:16.129109 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.129041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6fk\" (UniqueName: \"kubernetes.io/projected/8a73aae9-4d19-4b1f-b797-caa2488003d2-kube-api-access-tb6fk\") pod \"dns-operator-controller-manager-864dd78c4c-55wfw\" (UID: \"8a73aae9-4d19-4b1f-b797-caa2488003d2\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:16.176766 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.176736 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:16.312746 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.312607 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw"] Jul 08 13:29:16.315537 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:16.315504 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a73aae9_4d19_4b1f_b797_caa2488003d2.slice/crio-d326391d70303ef70d2bfa98a62cc817a9447aa798155098ce84bbb0048caa84 WatchSource:0}: Error finding container d326391d70303ef70d2bfa98a62cc817a9447aa798155098ce84bbb0048caa84: Status 404 returned error can't find the container with id d326391d70303ef70d2bfa98a62cc817a9447aa798155098ce84bbb0048caa84 Jul 08 13:29:16.946953 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:16.946914 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" event={"ID":"8a73aae9-4d19-4b1f-b797-caa2488003d2","Type":"ContainerStarted","Data":"d326391d70303ef70d2bfa98a62cc817a9447aa798155098ce84bbb0048caa84"} Jul 08 13:29:18.374345 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.374304 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n"] Jul 08 13:29:18.378258 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.378234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:18.380701 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.380681 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-72q2p\"" Jul 08 13:29:18.389576 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.389551 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n"] Jul 08 13:29:18.523120 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.523080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv28d\" (UniqueName: \"kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d\") pod \"limitador-operator-controller-manager-859d7b78bd-42l2n\" (UID: \"7d1b0fde-c32f-451d-8dab-de185028cdaf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:18.623762 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.623725 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv28d\" (UniqueName: \"kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d\") pod \"limitador-operator-controller-manager-859d7b78bd-42l2n\" (UID: \"7d1b0fde-c32f-451d-8dab-de185028cdaf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:18.632329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.632252 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv28d\" (UniqueName: \"kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d\") pod \"limitador-operator-controller-manager-859d7b78bd-42l2n\" (UID: \"7d1b0fde-c32f-451d-8dab-de185028cdaf\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:18.693267 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:18.693233 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:19.937407 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.937378 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n"] Jul 08 13:29:19.940007 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:19.939978 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1b0fde_c32f_451d_8dab_de185028cdaf.slice/crio-7471f12ab59bd6cf213446901eefd3bd428bcb9d2108705e9eed9b793fc19add WatchSource:0}: Error finding container 7471f12ab59bd6cf213446901eefd3bd428bcb9d2108705e9eed9b793fc19add: Status 404 returned error can't find the container with id 7471f12ab59bd6cf213446901eefd3bd428bcb9d2108705e9eed9b793fc19add Jul 08 13:29:19.966486 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.966455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" event={"ID":"7d1b0fde-c32f-451d-8dab-de185028cdaf","Type":"ContainerStarted","Data":"7471f12ab59bd6cf213446901eefd3bd428bcb9d2108705e9eed9b793fc19add"} Jul 08 13:29:19.968151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.968118 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" event={"ID":"8a73aae9-4d19-4b1f-b797-caa2488003d2","Type":"ContainerStarted","Data":"5cdc5a8238597f741ac04299cbc524569dd432b7a2996d6dac8d009ae4d797d3"} Jul 08 13:29:19.968404 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.968344 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:19.970368 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.970338 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" event={"ID":"0f6dbadb-425e-4364-b430-172259867091","Type":"ContainerStarted","Data":"d12a3cf9781c597f628de33467925a2d720a9358da1dd2f4c173ab6c496f702d"} Jul 08 13:29:19.970530 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.970513 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:19.990775 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:19.990725 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" podStartSLOduration=1.508824449 podStartE2EDuration="4.990707602s" totalImagesPullingTime="3.481883153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:19.988793481 +0000 UTC m=+398.482513291" watchObservedRunningTime="2026-07-08 13:29:19.990707602 +0000 UTC m=+398.484427455" Jul 08 13:29:20.008836 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:20.008785 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" podStartSLOduration=1.651486462 podStartE2EDuration="8.008769784s" totalImagesPullingTime="6.357283322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:20.007040538 +0000 UTC m=+398.500760378" watchObservedRunningTime="2026-07-08 13:29:20.008769784 +0000 UTC m=+398.502489572" Jul 08 13:29:21.923744 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:21.923705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-tzvx6" Jul 08 13:29:22.990057 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:22.990016 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" event={"ID":"7d1b0fde-c32f-451d-8dab-de185028cdaf","Type":"ContainerStarted","Data":"84b2a7933f3dcb6150784393aa02e9bdb11c61ce7a5d3bb17717d7113b4f56f2"} Jul 08 13:29:22.990493 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:22.990167 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:23.008436 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:23.008390 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" podStartSLOduration=2.88630573 podStartE2EDuration="5.008374711s" totalImagesPullingTime="2.122068981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:23.006509733 +0000 UTC m=+401.500229523" watchObservedRunningTime="2026-07-08 13:29:23.008374711 +0000 UTC m=+401.502094500" Jul 08 13:29:30.979521 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:30.979490 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:30.979974 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:30.979563 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-55wfw" Jul 08 13:29:32.885532 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.885497 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz"] Jul 08 13:29:32.885935 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.885686 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" podUID="0f6dbadb-425e-4364-b430-172259867091" containerName="manager" containerID="cri-o://d12a3cf9781c597f628de33467925a2d720a9358da1dd2f4c173ab6c496f702d" gracePeriod=2 Jul 08 13:29:32.898851 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.898825 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz"] Jul 08 13:29:32.910704 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.910679 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n"] Jul 08 13:29:32.910886 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.910866 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" containerName="manager" containerID="cri-o://84b2a7933f3dcb6150784393aa02e9bdb11c61ce7a5d3bb17717d7113b4f56f2" gracePeriod=2 Jul 08 13:29:32.912978 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.912939 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:32.913111 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.912936 2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0f6dbadb-425e-4364-b430-172259867091\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-279mz\": pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:32.915146 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.915117 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-42l2n\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:32.916694 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.916672 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:29:32.917228 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.917211 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f6dbadb-425e-4364-b430-172259867091" containerName="manager" Jul 08 13:29:32.917318 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.917230 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f6dbadb-425e-4364-b430-172259867091" containerName="manager" Jul 08 13:29:32.917365 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.917357 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f6dbadb-425e-4364-b430-172259867091" containerName="manager" Jul 08 13:29:32.917398 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.917367 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="0f6dbadb-425e-4364-b430-172259867091" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:32.920429 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.920411 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n"] Jul 08 13:29:32.920522 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.920489 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:32.922728 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.922705 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-42l2n\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:32.924899 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.924873 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="0f6dbadb-425e-4364-b430-172259867091" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:32.934985 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.934953 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:29:32.938493 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.938463 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l82zd\" (UniqueName: \"kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:32.938621 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.938543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:32.941107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.941087 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg"] Jul 08 13:29:32.941455 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.941441 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" containerName="manager" Jul 08 13:29:32.941455 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.941456 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" containerName="manager" Jul 08 13:29:32.941599 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.941526 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" containerName="manager" Jul 08 13:29:32.944578 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.944549 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:32.955697 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.955659 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-42l2n\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:32.957574 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:32.957549 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg"] Jul 08 13:29:32.957866 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:32.957826 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="0f6dbadb-425e-4364-b430-172259867091" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:33.038995 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.038966 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:33.039118 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.038968 2576 generic.go:356] "Generic (PLEG): container finished" podID="0f6dbadb-425e-4364-b430-172259867091" containerID="d12a3cf9781c597f628de33467925a2d720a9358da1dd2f4c173ab6c496f702d" exitCode=0 Jul 08 13:29:33.039118 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.039067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l82zd\" (UniqueName: \"kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:33.039118 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.039110 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fmd\" (UniqueName: \"kubernetes.io/projected/9e8c574e-e82c-47a0-b0ba-a54de614dfd8-kube-api-access-m7fmd\") pod \"limitador-operator-controller-manager-859d7b78bd-xfxdg\" (UID: \"9e8c574e-e82c-47a0-b0ba-a54de614dfd8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:33.039441 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.039414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:33.041045 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.041020 2576 generic.go:356] "Generic (PLEG): container finished" podID="7d1b0fde-c32f-451d-8dab-de185028cdaf" containerID="84b2a7933f3dcb6150784393aa02e9bdb11c61ce7a5d3bb17717d7113b4f56f2" exitCode=0 Jul 08 13:29:33.050816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.050597 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l82zd\" (UniqueName: \"kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-9g82l\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:33.140107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.140072 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fmd\" (UniqueName: \"kubernetes.io/projected/9e8c574e-e82c-47a0-b0ba-a54de614dfd8-kube-api-access-m7fmd\") pod \"limitador-operator-controller-manager-859d7b78bd-xfxdg\" (UID: \"9e8c574e-e82c-47a0-b0ba-a54de614dfd8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:33.150977 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.150950 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fmd\" (UniqueName: \"kubernetes.io/projected/9e8c574e-e82c-47a0-b0ba-a54de614dfd8-kube-api-access-m7fmd\") pod \"limitador-operator-controller-manager-859d7b78bd-xfxdg\" (UID: \"9e8c574e-e82c-47a0-b0ba-a54de614dfd8\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:33.173627 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.173603 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:33.175597 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.175579 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:33.175901 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:33.175879 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-42l2n\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:33.180810 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:33.180787 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="0f6dbadb-425e-4364-b430-172259867091" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:33.182871 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:33.182853 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-42l2n\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:33.184797 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:29:33.184779 2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-279mz\" is forbidden: User \"system:node:ip-10-0-138-212.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-212.ec2.internal' and this object" podUID="0f6dbadb-425e-4364-b430-172259867091" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:33.241055 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.241022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume\") pod \"0f6dbadb-425e-4364-b430-172259867091\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " Jul 08 13:29:33.241262 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.241144 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dllhg\" (UniqueName: \"kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg\") pod \"0f6dbadb-425e-4364-b430-172259867091\" (UID: \"0f6dbadb-425e-4364-b430-172259867091\") " Jul 08 13:29:33.241262 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.241217 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cv28d\" (UniqueName: \"kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d\") pod \"7d1b0fde-c32f-451d-8dab-de185028cdaf\" (UID: \"7d1b0fde-c32f-451d-8dab-de185028cdaf\") " Jul 08 13:29:33.241548 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.241522 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0f6dbadb-425e-4364-b430-172259867091" (UID: "0f6dbadb-425e-4364-b430-172259867091"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 13:29:33.243324 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.243299 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg" (OuterVolumeSpecName: "kube-api-access-dllhg") pod "0f6dbadb-425e-4364-b430-172259867091" (UID: "0f6dbadb-425e-4364-b430-172259867091"). InnerVolumeSpecName "kube-api-access-dllhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:29:33.243423 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.243330 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d" (OuterVolumeSpecName: "kube-api-access-cv28d") pod "7d1b0fde-c32f-451d-8dab-de185028cdaf" (UID: "7d1b0fde-c32f-451d-8dab-de185028cdaf"). InnerVolumeSpecName "kube-api-access-cv28d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:29:33.320241 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.320210 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:33.326258 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.326234 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:33.342709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.342687 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dllhg\" (UniqueName: \"kubernetes.io/projected/0f6dbadb-425e-4364-b430-172259867091-kube-api-access-dllhg\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:29:33.342799 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.342714 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv28d\" (UniqueName: \"kubernetes.io/projected/7d1b0fde-c32f-451d-8dab-de185028cdaf-kube-api-access-cv28d\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:29:33.342799 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.342726 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0f6dbadb-425e-4364-b430-172259867091-extensions-socket-volume\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:29:33.471997 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.471964 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:29:33.475440 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:33.475413 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9046b11e_257f_4316_90e9_daec9c4e760f.slice/crio-722f95fd6a570a3eb4912d7936b4c7a4de572fdef3536bd6df2eed5818c0d686 WatchSource:0}: Error finding container 722f95fd6a570a3eb4912d7936b4c7a4de572fdef3536bd6df2eed5818c0d686: Status 404 returned error can't find the container with id 722f95fd6a570a3eb4912d7936b4c7a4de572fdef3536bd6df2eed5818c0d686 Jul 08 13:29:33.497777 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:33.497750 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg"] Jul 08 13:29:33.500198 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:29:33.500149 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8c574e_e82c_47a0_b0ba_a54de614dfd8.slice/crio-18c983dab404fc758c1c763693bd4aa60949e734703192bdbb3a5a73f26631d8 WatchSource:0}: Error finding container 18c983dab404fc758c1c763693bd4aa60949e734703192bdbb3a5a73f26631d8: Status 404 returned error can't find the container with id 18c983dab404fc758c1c763693bd4aa60949e734703192bdbb3a5a73f26631d8 Jul 08 13:29:34.048346 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.048309 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f6dbadb-425e-4364-b430-172259867091" path="/var/lib/kubelet/pods/0f6dbadb-425e-4364-b430-172259867091/volumes" Jul 08 13:29:34.048734 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.048702 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d1b0fde-c32f-451d-8dab-de185028cdaf" path="/var/lib/kubelet/pods/7d1b0fde-c32f-451d-8dab-de185028cdaf/volumes" Jul 08 13:29:34.050138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.050113 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-279mz" Jul 08 13:29:34.050282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.050139 2576 scope.go:117] "RemoveContainer" containerID="d12a3cf9781c597f628de33467925a2d720a9358da1dd2f4c173ab6c496f702d" Jul 08 13:29:34.051816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.051797 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-42l2n" Jul 08 13:29:34.053725 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.053695 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" event={"ID":"9046b11e-257f-4316-90e9-daec9c4e760f","Type":"ContainerStarted","Data":"30cae8f3d0b7cd31ce3d105537acd20e516cacbfaa67e2942724413eefd92efb"} Jul 08 13:29:34.053896 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.053871 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" event={"ID":"9046b11e-257f-4316-90e9-daec9c4e760f","Type":"ContainerStarted","Data":"722f95fd6a570a3eb4912d7936b4c7a4de572fdef3536bd6df2eed5818c0d686"} Jul 08 13:29:34.054014 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.053934 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:34.055654 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.055626 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" event={"ID":"9e8c574e-e82c-47a0-b0ba-a54de614dfd8","Type":"ContainerStarted","Data":"e81474bd7666bb475ae2dcf500ad7f16bd2ac34d725bf9a3a40e85730cf8dabd"} Jul 08 13:29:34.055755 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.055679 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" event={"ID":"9e8c574e-e82c-47a0-b0ba-a54de614dfd8","Type":"ContainerStarted","Data":"18c983dab404fc758c1c763693bd4aa60949e734703192bdbb3a5a73f26631d8"} Jul 08 13:29:34.055848 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.055793 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:29:34.061524 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.061504 2576 scope.go:117] "RemoveContainer" containerID="84b2a7933f3dcb6150784393aa02e9bdb11c61ce7a5d3bb17717d7113b4f56f2" Jul 08 13:29:34.076046 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.076002 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" podStartSLOduration=2.075989567 podStartE2EDuration="2.075989567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:34.075387857 +0000 UTC m=+412.569107647" watchObservedRunningTime="2026-07-08 13:29:34.075989567 +0000 UTC m=+412.569709355" Jul 08 13:29:34.095572 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:34.095531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" podStartSLOduration=2.095516937 podStartE2EDuration="2.095516937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:29:34.094591177 +0000 UTC m=+412.588310959" watchObservedRunningTime="2026-07-08 13:29:34.095516937 +0000 UTC m=+412.589236726" Jul 08 13:29:45.064247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:45.064210 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:29:45.064667 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:29:45.064303 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xfxdg" Jul 08 13:30:06.051608 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.051518 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:30:06.052208 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.052154 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" podUID="9046b11e-257f-4316-90e9-daec9c4e760f" containerName="manager" containerID="cri-o://30cae8f3d0b7cd31ce3d105537acd20e516cacbfaa67e2942724413eefd92efb" gracePeriod=10 Jul 08 13:30:06.205956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.205922 2576 generic.go:356] "Generic (PLEG): container finished" podID="9046b11e-257f-4316-90e9-daec9c4e760f" containerID="30cae8f3d0b7cd31ce3d105537acd20e516cacbfaa67e2942724413eefd92efb" exitCode=0 Jul 08 13:30:06.206112 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.206000 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" event={"ID":"9046b11e-257f-4316-90e9-daec9c4e760f","Type":"ContainerDied","Data":"30cae8f3d0b7cd31ce3d105537acd20e516cacbfaa67e2942724413eefd92efb"} Jul 08 13:30:06.316250 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.316165 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:30:06.427777 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.427740 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume\") pod \"9046b11e-257f-4316-90e9-daec9c4e760f\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " Jul 08 13:30:06.427935 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.427850 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l82zd\" (UniqueName: \"kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd\") pod \"9046b11e-257f-4316-90e9-daec9c4e760f\" (UID: \"9046b11e-257f-4316-90e9-daec9c4e760f\") " Jul 08 13:30:06.428241 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.428206 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9046b11e-257f-4316-90e9-daec9c4e760f" (UID: "9046b11e-257f-4316-90e9-daec9c4e760f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 13:30:06.429902 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.429877 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd" (OuterVolumeSpecName: "kube-api-access-l82zd") pod "9046b11e-257f-4316-90e9-daec9c4e760f" (UID: "9046b11e-257f-4316-90e9-daec9c4e760f"). InnerVolumeSpecName "kube-api-access-l82zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:06.528795 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.528761 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l82zd\" (UniqueName: \"kubernetes.io/projected/9046b11e-257f-4316-90e9-daec9c4e760f-kube-api-access-l82zd\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:06.528795 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:06.528788 2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9046b11e-257f-4316-90e9-daec9c4e760f-extensions-socket-volume\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:07.212228 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:07.212197 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" Jul 08 13:30:07.212228 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:07.212211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l" event={"ID":"9046b11e-257f-4316-90e9-daec9c4e760f","Type":"ContainerDied","Data":"722f95fd6a570a3eb4912d7936b4c7a4de572fdef3536bd6df2eed5818c0d686"} Jul 08 13:30:07.212693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:07.212272 2576 scope.go:117] "RemoveContainer" containerID="30cae8f3d0b7cd31ce3d105537acd20e516cacbfaa67e2942724413eefd92efb" Jul 08 13:30:07.235612 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:07.235588 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:30:07.239454 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:07.239432 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-9g82l"] Jul 08 13:30:08.042984 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:08.042949 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9046b11e-257f-4316-90e9-daec9c4e760f" path="/var/lib/kubelet/pods/9046b11e-257f-4316-90e9-daec9c4e760f/volumes" Jul 08 13:30:22.436433 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.436388 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw"] Jul 08 13:30:22.436972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.436958 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9046b11e-257f-4316-90e9-daec9c4e760f" containerName="manager" Jul 08 13:30:22.437027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.436976 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9046b11e-257f-4316-90e9-daec9c4e760f" containerName="manager" Jul 08 13:30:22.437067 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.437056 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9046b11e-257f-4316-90e9-daec9c4e760f" containerName="manager" Jul 08 13:30:22.441745 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.441723 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.444120 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.444101 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 13:30:22.444245 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.444152 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8mtsb\"" Jul 08 13:30:22.450787 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.450758 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw"] Jul 08 13:30:22.462149 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462276 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf8sc\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-kube-api-access-cf8sc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462276 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462255 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462285 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30918a42-4d90-4bff-83ce-b2facafdd2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462357 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462512 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462562 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.462615 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.462600 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563041 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30918a42-4d90-4bff-83ce-b2facafdd2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563041 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563045 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563085 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563113 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563135 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563232 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf8sc\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-kube-api-access-cf8sc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563305 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563299 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563718 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563718 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563668 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.563895 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.563872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30918a42-4d90-4bff-83ce-b2facafdd2b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.564282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.564258 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.564501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.564482 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.565951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.565917 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.566153 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.566137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.571078 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.571044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.571265 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.571120 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf8sc\" (UniqueName: \"kubernetes.io/projected/30918a42-4d90-4bff-83ce-b2facafdd2b8-kube-api-access-cf8sc\") pod \"maas-default-gateway-openshift-default-5896d4dd49-fn5sw\" (UID: \"30918a42-4d90-4bff-83ce-b2facafdd2b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.753821 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.753719 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:22.896420 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:22.896394 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw"] Jul 08 13:30:22.898461 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:22.898437 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30918a42_4d90_4bff_83ce_b2facafdd2b8.slice/crio-06a3e22f40e2ed578a4cdf8413b8042d97b3625c4ee2f44fbfbb90ac9db2f4af WatchSource:0}: Error finding container 06a3e22f40e2ed578a4cdf8413b8042d97b3625c4ee2f44fbfbb90ac9db2f4af: Status 404 returned error can't find the container with id 06a3e22f40e2ed578a4cdf8413b8042d97b3625c4ee2f44fbfbb90ac9db2f4af Jul 08 13:30:23.287249 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:23.287213 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" event={"ID":"30918a42-4d90-4bff-83ce-b2facafdd2b8","Type":"ContainerStarted","Data":"06a3e22f40e2ed578a4cdf8413b8042d97b3625c4ee2f44fbfbb90ac9db2f4af"} Jul 08 13:30:25.321728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:25.321691 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:30:25.321987 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:25.321764 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:30:25.321987 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:25.321793 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:30:26.304130 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:26.304088 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" event={"ID":"30918a42-4d90-4bff-83ce-b2facafdd2b8","Type":"ContainerStarted","Data":"9352a322a813e0a529661a617cff8b7f1ee817a0e66230f8161f8236336464d1"} Jul 08 13:30:26.326288 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:26.326237 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" podStartSLOduration=1.905263739 podStartE2EDuration="4.326218927s" totalImagesPullingTime="2.420955188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:26.323729757 +0000 UTC m=+464.817449561" watchObservedRunningTime="2026-07-08 13:30:26.326218927 +0000 UTC m=+464.819938716" Jul 08 13:30:26.754730 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:26.754687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:26.759431 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:26.759407 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:27.309427 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.309396 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:27.310427 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.310409 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:30:27.381611 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.381579 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:27.387750 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.387714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.392720 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.392646 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 13:30:27.392970 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.392940 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:27.393090 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.393028 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4zqwg\"" Jul 08 13:30:27.413451 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.413421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.413451 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.413450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn8mm\" (UniqueName: \"kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.480704 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.480672 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:27.514504 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.514473 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.514504 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.514503 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn8mm\" (UniqueName: \"kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.515091 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.515070 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.523762 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.523730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn8mm\" (UniqueName: \"kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm\") pod \"limitador-limitador-64c94fd64f-6btbf\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.700030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.699996 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:27.854950 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:27.854926 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:27.857339 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:27.857316 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdabfaf_d0ac_4f3c_bcf7_b9db84d52135.slice/crio-904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6 WatchSource:0}: Error finding container 904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6: Status 404 returned error can't find the container with id 904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6 Jul 08 13:30:28.166059 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.166017 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:28.172311 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.172281 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:28.175103 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.175081 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-76cv9\"" Jul 08 13:30:28.178892 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.178868 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:28.221166 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.221133 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxq6r\" (UniqueName: \"kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r\") pod \"authorino-58cd955675-4k47r\" (UID: \"22002e51-5b4f-4a68-b2ab-98645de6a8fa\") " pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:28.315601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.315569 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" event={"ID":"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135","Type":"ContainerStarted","Data":"904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6"} Jul 08 13:30:28.322313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.322285 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxq6r\" (UniqueName: \"kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r\") pod \"authorino-58cd955675-4k47r\" (UID: \"22002e51-5b4f-4a68-b2ab-98645de6a8fa\") " pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:28.330228 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.330205 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxq6r\" (UniqueName: \"kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r\") pod \"authorino-58cd955675-4k47r\" (UID: \"22002e51-5b4f-4a68-b2ab-98645de6a8fa\") " pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:28.356363 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.356335 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:30:28.359716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.359695 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:28.366254 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.366233 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:30:28.423332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.423241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mx5s\" (UniqueName: \"kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s\") pod \"authorino-75d556bd77-flnc8\" (UID: \"7127973f-33cb-4a4f-b7cf-b2a230ace6f7\") " pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:28.483458 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.483431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:28.524163 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.524124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mx5s\" (UniqueName: \"kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s\") pod \"authorino-75d556bd77-flnc8\" (UID: \"7127973f-33cb-4a4f-b7cf-b2a230ace6f7\") " pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:28.532916 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.532859 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mx5s\" (UniqueName: \"kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s\") pod \"authorino-75d556bd77-flnc8\" (UID: \"7127973f-33cb-4a4f-b7cf-b2a230ace6f7\") " pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:28.631873 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.631843 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:28.635380 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:28.635335 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22002e51_5b4f_4a68_b2ab_98645de6a8fa.slice/crio-170636044f2aefd983aa47e525c3ad6c886d32f914bcf6ecf11bbae534ba339b WatchSource:0}: Error finding container 170636044f2aefd983aa47e525c3ad6c886d32f914bcf6ecf11bbae534ba339b: Status 404 returned error can't find the container with id 170636044f2aefd983aa47e525c3ad6c886d32f914bcf6ecf11bbae534ba339b Jul 08 13:30:28.670052 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.670017 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:28.842113 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:28.842054 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:30:28.847422 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:28.847388 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7127973f_33cb_4a4f_b7cf_b2a230ace6f7.slice/crio-1e8138753ce1dcf8bf1112c49bde0045489e67a42b1876acd8654603cc9edc45 WatchSource:0}: Error finding container 1e8138753ce1dcf8bf1112c49bde0045489e67a42b1876acd8654603cc9edc45: Status 404 returned error can't find the container with id 1e8138753ce1dcf8bf1112c49bde0045489e67a42b1876acd8654603cc9edc45 Jul 08 13:30:29.324966 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:29.324925 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-flnc8" event={"ID":"7127973f-33cb-4a4f-b7cf-b2a230ace6f7","Type":"ContainerStarted","Data":"1e8138753ce1dcf8bf1112c49bde0045489e67a42b1876acd8654603cc9edc45"} Jul 08 13:30:29.327247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:29.327198 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4k47r" event={"ID":"22002e51-5b4f-4a68-b2ab-98645de6a8fa","Type":"ContainerStarted","Data":"170636044f2aefd983aa47e525c3ad6c886d32f914bcf6ecf11bbae534ba339b"} Jul 08 13:30:33.354370 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.354336 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" event={"ID":"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135","Type":"ContainerStarted","Data":"6361a97f5c4e06978a6466255b1700dcf177ac674803a70374ee1b91a08a5865"} Jul 08 13:30:33.354817 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.354414 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:33.355791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.355767 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-flnc8" event={"ID":"7127973f-33cb-4a4f-b7cf-b2a230ace6f7","Type":"ContainerStarted","Data":"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5"} Jul 08 13:30:33.357067 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.357047 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4k47r" event={"ID":"22002e51-5b4f-4a68-b2ab-98645de6a8fa","Type":"ContainerStarted","Data":"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa"} Jul 08 13:30:33.374598 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.374555 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" podStartSLOduration=1.67576516 podStartE2EDuration="6.374541737s" totalImagesPullingTime="4.698776577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:33.373441547 +0000 UTC m=+471.867161357" watchObservedRunningTime="2026-07-08 13:30:33.374541737 +0000 UTC m=+471.868261581" Jul 08 13:30:33.388326 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.388277 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-4k47r" podStartSLOduration=1.411001371 podStartE2EDuration="5.388264154s" totalImagesPullingTime="3.977262783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:33.387354708 +0000 UTC m=+471.881074508" watchObservedRunningTime="2026-07-08 13:30:33.388264154 +0000 UTC m=+471.881983939" Jul 08 13:30:33.403308 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.403263 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-flnc8" podStartSLOduration=1.6349912519999998 podStartE2EDuration="5.403247804s" totalImagesPullingTime="3.768256552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:33.401369149 +0000 UTC m=+471.895088939" watchObservedRunningTime="2026-07-08 13:30:33.403247804 +0000 UTC m=+471.896967612" Jul 08 13:30:33.430754 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:33.430720 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:35.366079 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:35.366042 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-4k47r" podUID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" containerName="authorino" containerID="cri-o://186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa" gracePeriod=30 Jul 08 13:30:35.631334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:35.631271 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:35.688324 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:35.688297 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sxq6r\" (UniqueName: \"kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r\") pod \"22002e51-5b4f-4a68-b2ab-98645de6a8fa\" (UID: \"22002e51-5b4f-4a68-b2ab-98645de6a8fa\") " Jul 08 13:30:35.690343 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:35.690312 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r" (OuterVolumeSpecName: "kube-api-access-sxq6r") pod "22002e51-5b4f-4a68-b2ab-98645de6a8fa" (UID: "22002e51-5b4f-4a68-b2ab-98645de6a8fa"). InnerVolumeSpecName "kube-api-access-sxq6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:35.789147 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:35.789116 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxq6r\" (UniqueName: \"kubernetes.io/projected/22002e51-5b4f-4a68-b2ab-98645de6a8fa-kube-api-access-sxq6r\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:36.372012 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.371923 2576 generic.go:356] "Generic (PLEG): container finished" podID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" containerID="186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa" exitCode=0 Jul 08 13:30:36.372012 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.371977 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-4k47r" Jul 08 13:30:36.372462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.372009 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4k47r" event={"ID":"22002e51-5b4f-4a68-b2ab-98645de6a8fa","Type":"ContainerDied","Data":"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa"} Jul 08 13:30:36.372462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.372055 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-4k47r" event={"ID":"22002e51-5b4f-4a68-b2ab-98645de6a8fa","Type":"ContainerDied","Data":"170636044f2aefd983aa47e525c3ad6c886d32f914bcf6ecf11bbae534ba339b"} Jul 08 13:30:36.372462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.372077 2576 scope.go:117] "RemoveContainer" containerID="186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa" Jul 08 13:30:36.380796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.380775 2576 scope.go:117] "RemoveContainer" containerID="186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa" Jul 08 13:30:36.381058 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:30:36.381039 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa\": container with ID starting with 186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa not found: ID does not exist" containerID="186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa" Jul 08 13:30:36.381107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.381067 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa"} err="failed to get container status \"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa\": rpc error: code = NotFound desc = could not find container \"186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa\": container with ID starting with 186e99e793d3055da2a4e6fdbe400faec71287f0b6cfb4a9e480d84d493afdaa not found: ID does not exist" Jul 08 13:30:36.388227 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.388205 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:36.394003 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:36.393983 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-4k47r"] Jul 08 13:30:38.043427 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:38.043395 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" path="/var/lib/kubelet/pods/22002e51-5b4f-4a68-b2ab-98645de6a8fa/volumes" Jul 08 13:30:41.854627 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:41.854591 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:41.855071 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:41.854758 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" podUID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" containerName="limitador" containerID="cri-o://6361a97f5c4e06978a6466255b1700dcf177ac674803a70374ee1b91a08a5865" gracePeriod=30 Jul 08 13:30:41.855443 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:41.855425 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:42.403032 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.403005 2576 generic.go:356] "Generic (PLEG): container finished" podID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" containerID="6361a97f5c4e06978a6466255b1700dcf177ac674803a70374ee1b91a08a5865" exitCode=0 Jul 08 13:30:42.403153 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.403078 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" event={"ID":"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135","Type":"ContainerDied","Data":"6361a97f5c4e06978a6466255b1700dcf177ac674803a70374ee1b91a08a5865"} Jul 08 13:30:42.403153 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.403129 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" event={"ID":"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135","Type":"ContainerDied","Data":"904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6"} Jul 08 13:30:42.403153 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.403147 2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="904fe00bc07684aa9961d9d8c20771f9c416f6ccfde41bef0ef0cb88700282a6" Jul 08 13:30:42.416755 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.416704 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:42.549223 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.549192 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file\") pod \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " Jul 08 13:30:42.549399 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.549288 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dn8mm\" (UniqueName: \"kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm\") pod \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\" (UID: \"ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135\") " Jul 08 13:30:42.549543 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.549519 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file" (OuterVolumeSpecName: "config-file") pod "ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" (UID: "ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:30:42.551332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.551304 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm" (OuterVolumeSpecName: "kube-api-access-dn8mm") pod "ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" (UID: "ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135"). InnerVolumeSpecName "kube-api-access-dn8mm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:42.650138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.650103 2576 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-config-file\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:42.650138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:42.650133 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn8mm\" (UniqueName: \"kubernetes.io/projected/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135-kube-api-access-dn8mm\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:43.407760 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.407722 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-6btbf" Jul 08 13:30:43.430693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.430666 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:43.433149 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.433124 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-6btbf"] Jul 08 13:30:43.970596 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.970563 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t"] Jul 08 13:30:43.970935 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.970924 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" containerName="limitador" Jul 08 13:30:43.970979 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.970938 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" containerName="limitador" Jul 08 13:30:43.970979 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.970948 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" containerName="authorino" Jul 08 13:30:43.970979 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.970953 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" containerName="authorino" Jul 08 13:30:43.971062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.971014 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="22002e51-5b4f-4a68-b2ab-98645de6a8fa" containerName="authorino" Jul 08 13:30:43.971062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.971021 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" containerName="limitador" Jul 08 13:30:43.974127 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.974111 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:43.976512 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.976484 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 08 13:30:43.976644 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.976491 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 13:30:43.976644 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.976572 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zxl8v\"" Jul 08 13:30:43.977878 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.977845 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 13:30:43.980042 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:43.980023 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t"] Jul 08 13:30:44.042947 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.042913 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135" path="/var/lib/kubelet/pods/ccdabfaf-d0ac-4f3c-bcf7-b9db84d52135/volumes" Jul 08 13:30:44.062971 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.062947 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv27j\" (UniqueName: \"kubernetes.io/projected/632d33c3-1986-4a50-8ff0-d59649c68b65-kube-api-access-jv27j\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.063077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.063000 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/632d33c3-1986-4a50-8ff0-d59649c68b65-data\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.163835 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.163800 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/632d33c3-1986-4a50-8ff0-d59649c68b65-data\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.164020 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.163874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv27j\" (UniqueName: \"kubernetes.io/projected/632d33c3-1986-4a50-8ff0-d59649c68b65-kube-api-access-jv27j\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.164168 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.164147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/632d33c3-1986-4a50-8ff0-d59649c68b65-data\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.172094 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.172069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv27j\" (UniqueName: \"kubernetes.io/projected/632d33c3-1986-4a50-8ff0-d59649c68b65-kube-api-access-jv27j\") pod \"postgres-69fb4594cf-pbd7t\" (UID: \"632d33c3-1986-4a50-8ff0-d59649c68b65\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.286661 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.286583 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:44.438684 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:44.438662 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t"] Jul 08 13:30:44.440813 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:44.440775 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632d33c3_1986_4a50_8ff0_d59649c68b65.slice/crio-033103cdb6893cd8f031009a7d93dc08f73d4f2427f8ca5c547b9345f00a3b5e WatchSource:0}: Error finding container 033103cdb6893cd8f031009a7d93dc08f73d4f2427f8ca5c547b9345f00a3b5e: Status 404 returned error can't find the container with id 033103cdb6893cd8f031009a7d93dc08f73d4f2427f8ca5c547b9345f00a3b5e Jul 08 13:30:45.422933 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:45.422895 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" event={"ID":"632d33c3-1986-4a50-8ff0-d59649c68b65","Type":"ContainerStarted","Data":"033103cdb6893cd8f031009a7d93dc08f73d4f2427f8ca5c547b9345f00a3b5e"} Jul 08 13:30:49.445743 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:49.445697 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" event={"ID":"632d33c3-1986-4a50-8ff0-d59649c68b65","Type":"ContainerStarted","Data":"9adc4106723900f3ca9679c320ca43f8d8aec85cd8d93e4f057f3318afef4084"} Jul 08 13:30:49.446139 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:49.445769 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:49.465238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:49.465187 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" podStartSLOduration=1.546636342 podStartE2EDuration="6.465152891s" totalImagesPullingTime="4.918516549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:49.463590437 +0000 UTC m=+487.957310226" watchObservedRunningTime="2026-07-08 13:30:49.465152891 +0000 UTC m=+487.958872681" Jul 08 13:30:55.480908 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:55.480874 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-pbd7t" Jul 08 13:30:56.005998 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.005963 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xmpjl"] Jul 08 13:30:56.011799 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.011774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.019215 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.019188 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xmpjl"] Jul 08 13:30:56.176809 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.176779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4shr\" (UniqueName: \"kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr\") pod \"authorino-5d4b67c446-xmpjl\" (UID: \"0a233d77-6c77-4bc4-94fc-d7f86ff1c672\") " pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.244384 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.244352 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xmpjl"] Jul 08 13:30:56.244646 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:30:56.244627 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-q4shr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-xmpjl" podUID="0a233d77-6c77-4bc4-94fc-d7f86ff1c672" Jul 08 13:30:56.269724 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.269649 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-f485f7546-9n5pf"] Jul 08 13:30:56.273727 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.273707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.277439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.277399 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4shr\" (UniqueName: \"kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr\") pod \"authorino-5d4b67c446-xmpjl\" (UID: \"0a233d77-6c77-4bc4-94fc-d7f86ff1c672\") " pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.279108 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.279088 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-f485f7546-9n5pf"] Jul 08 13:30:56.290679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.290653 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4shr\" (UniqueName: \"kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr\") pod \"authorino-5d4b67c446-xmpjl\" (UID: \"0a233d77-6c77-4bc4-94fc-d7f86ff1c672\") " pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.361579 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.361549 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f485f7546-9n5pf"] Jul 08 13:30:56.361827 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:30:56.361808 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-vtmn7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-f485f7546-9n5pf" podUID="e0722ffd-4a39-474b-a4a8-dfd713eadc84" Jul 08 13:30:56.378677 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.378644 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtmn7\" (UniqueName: \"kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7\") pod \"authorino-f485f7546-9n5pf\" (UID: \"e0722ffd-4a39-474b-a4a8-dfd713eadc84\") " pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.388166 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.388138 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:30:56.391927 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.391911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.394323 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.394305 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 13:30:56.398414 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.398392 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:30:56.477135 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.477104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.477332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.477104 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.479513 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.479494 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtmn7\" (UniqueName: \"kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7\") pod \"authorino-f485f7546-9n5pf\" (UID: \"e0722ffd-4a39-474b-a4a8-dfd713eadc84\") " pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.479632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.479524 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjcl\" (UniqueName: \"kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.479632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.479556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.484216 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.484196 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.487282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.487258 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtmn7\" (UniqueName: \"kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7\") pod \"authorino-f485f7546-9n5pf\" (UID: \"e0722ffd-4a39-474b-a4a8-dfd713eadc84\") " pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:56.487282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.487271 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:56.579938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.579851 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q4shr\" (UniqueName: \"kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr\") pod \"0a233d77-6c77-4bc4-94fc-d7f86ff1c672\" (UID: \"0a233d77-6c77-4bc4-94fc-d7f86ff1c672\") " Jul 08 13:30:56.579938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.579912 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtmn7\" (UniqueName: \"kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7\") pod \"e0722ffd-4a39-474b-a4a8-dfd713eadc84\" (UID: \"e0722ffd-4a39-474b-a4a8-dfd713eadc84\") " Jul 08 13:30:56.580208 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.580034 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjcl\" (UniqueName: \"kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.580208 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.580071 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.582248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.582216 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7" (OuterVolumeSpecName: "kube-api-access-vtmn7") pod "e0722ffd-4a39-474b-a4a8-dfd713eadc84" (UID: "e0722ffd-4a39-474b-a4a8-dfd713eadc84"). InnerVolumeSpecName "kube-api-access-vtmn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:56.582361 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.582251 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr" (OuterVolumeSpecName: "kube-api-access-q4shr") pod "0a233d77-6c77-4bc4-94fc-d7f86ff1c672" (UID: "0a233d77-6c77-4bc4-94fc-d7f86ff1c672"). InnerVolumeSpecName "kube-api-access-q4shr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:56.582535 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.582513 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.588988 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.588969 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjcl\" (UniqueName: \"kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl\") pod \"authorino-86f8d7798b-x68jj\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.681224 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.681160 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4shr\" (UniqueName: \"kubernetes.io/projected/0a233d77-6c77-4bc4-94fc-d7f86ff1c672-kube-api-access-q4shr\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:56.681224 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.681219 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtmn7\" (UniqueName: \"kubernetes.io/projected/e0722ffd-4a39-474b-a4a8-dfd713eadc84-kube-api-access-vtmn7\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:56.702028 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.702000 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:30:56.838270 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:56.838192 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:30:56.840554 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:30:56.840524 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod63ff4ef1_f14a_403f_92b7_6cff0a24efb0.slice/crio-30ba6735c57484a8edc9a01f40f3efd6ce34c9ad65f0b4f196ac9f3e13f56482 WatchSource:0}: Error finding container 30ba6735c57484a8edc9a01f40f3efd6ce34c9ad65f0b4f196ac9f3e13f56482: Status 404 returned error can't find the container with id 30ba6735c57484a8edc9a01f40f3efd6ce34c9ad65f0b4f196ac9f3e13f56482 Jul 08 13:30:57.482951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.482920 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-xmpjl" Jul 08 13:30:57.482951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.482942 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-f485f7546-9n5pf" Jul 08 13:30:57.482951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.482921 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86f8d7798b-x68jj" event={"ID":"63ff4ef1-f14a-403f-92b7-6cff0a24efb0","Type":"ContainerStarted","Data":"30ba6735c57484a8edc9a01f40f3efd6ce34c9ad65f0b4f196ac9f3e13f56482"} Jul 08 13:30:57.518872 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.518838 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-f485f7546-9n5pf"] Jul 08 13:30:57.520733 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.520700 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-f485f7546-9n5pf"] Jul 08 13:30:57.539383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.539351 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xmpjl"] Jul 08 13:30:57.540915 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:57.540893 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-xmpjl"] Jul 08 13:30:58.043162 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.043130 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a233d77-6c77-4bc4-94fc-d7f86ff1c672" path="/var/lib/kubelet/pods/0a233d77-6c77-4bc4-94fc-d7f86ff1c672/volumes" Jul 08 13:30:58.043388 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.043375 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0722ffd-4a39-474b-a4a8-dfd713eadc84" path="/var/lib/kubelet/pods/e0722ffd-4a39-474b-a4a8-dfd713eadc84/volumes" Jul 08 13:30:58.489184 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.489133 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86f8d7798b-x68jj" event={"ID":"63ff4ef1-f14a-403f-92b7-6cff0a24efb0","Type":"ContainerStarted","Data":"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c"} Jul 08 13:30:58.504617 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.504573 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-86f8d7798b-x68jj" podStartSLOduration=1.7988833579999999 podStartE2EDuration="2.504548504s" totalImagesPullingTime="705.665146ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:30:58.504313623 +0000 UTC m=+496.998033412" watchObservedRunningTime="2026-07-08 13:30:58.504548504 +0000 UTC m=+496.998274858" Jul 08 13:30:58.531876 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.531846 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:30:58.532266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.532038 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-flnc8" podUID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" containerName="authorino" containerID="cri-o://0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5" gracePeriod=30 Jul 08 13:30:58.827165 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:58.827135 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:59.005505 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.005470 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mx5s\" (UniqueName: \"kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s\") pod \"7127973f-33cb-4a4f-b7cf-b2a230ace6f7\" (UID: \"7127973f-33cb-4a4f-b7cf-b2a230ace6f7\") " Jul 08 13:30:59.007553 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.007524 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s" (OuterVolumeSpecName: "kube-api-access-6mx5s") pod "7127973f-33cb-4a4f-b7cf-b2a230ace6f7" (UID: "7127973f-33cb-4a4f-b7cf-b2a230ace6f7"). InnerVolumeSpecName "kube-api-access-6mx5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:30:59.106985 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.106905 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mx5s\" (UniqueName: \"kubernetes.io/projected/7127973f-33cb-4a4f-b7cf-b2a230ace6f7-kube-api-access-6mx5s\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:30:59.495493 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.495457 2576 generic.go:356] "Generic (PLEG): container finished" podID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" containerID="0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5" exitCode=0 Jul 08 13:30:59.495672 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.495505 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-flnc8" Jul 08 13:30:59.495672 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.495539 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-flnc8" event={"ID":"7127973f-33cb-4a4f-b7cf-b2a230ace6f7","Type":"ContainerDied","Data":"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5"} Jul 08 13:30:59.495672 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.495577 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-flnc8" event={"ID":"7127973f-33cb-4a4f-b7cf-b2a230ace6f7","Type":"ContainerDied","Data":"1e8138753ce1dcf8bf1112c49bde0045489e67a42b1876acd8654603cc9edc45"} Jul 08 13:30:59.495672 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.495595 2576 scope.go:117] "RemoveContainer" containerID="0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5" Jul 08 13:30:59.504796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.504779 2576 scope.go:117] "RemoveContainer" containerID="0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5" Jul 08 13:30:59.505076 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:30:59.505058 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5\": container with ID starting with 0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5 not found: ID does not exist" containerID="0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5" Jul 08 13:30:59.505119 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.505085 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5"} err="failed to get container status \"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5\": rpc error: code = NotFound desc = could not find container \"0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5\": container with ID starting with 0ed1c181cb5ada332788607c79308c47dafe69aec9cf733b8c789727ae001ed5 not found: ID does not exist" Jul 08 13:30:59.517613 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.517590 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:30:59.522101 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:30:59.522082 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-flnc8"] Jul 08 13:31:00.043291 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:00.043258 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" path="/var/lib/kubelet/pods/7127973f-33cb-4a4f-b7cf-b2a230ace6f7/volumes" Jul 08 13:31:02.848529 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:02.848490 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:02.849050 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:02.849030 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" containerName="authorino" Jul 08 13:31:02.849134 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:02.849054 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" containerName="authorino" Jul 08 13:31:02.849222 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:02.849195 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7127973f-33cb-4a4f-b7cf-b2a230ace6f7" containerName="authorino" Jul 08 13:31:03.047310 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.047274 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:03.047660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.047629 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:31:03.047660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.047660 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.050380 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.050361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 13:31:03.051287 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.051265 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 13:31:03.051386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.051267 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gxfhm\"" Jul 08 13:31:03.081383 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.081351 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:31:03.081510 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.081453 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.242030 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.241993 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.242248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.242106 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wvd\" (UniqueName: \"kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.242248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.242135 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8l7\" (UniqueName: \"kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.242248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.242164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.342810 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.342775 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wvd\" (UniqueName: \"kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.342810 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.342812 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8l7\" (UniqueName: \"kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.343072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.342836 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.343072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.342886 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.345453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.345429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.345564 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.345481 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.351541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.351514 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wvd\" (UniqueName: \"kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd\") pod \"maas-controller-d5d4c5795-v88p8\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.351654 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.351619 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8l7\" (UniqueName: \"kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7\") pod \"maas-controller-7d7b4dd59-w5t9x\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.359478 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.359456 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:03.390587 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.390563 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:03.532500 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.532461 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:03.535682 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:31:03.535612 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86d6fa1_eed5_4586_a6eb_9ab7dda80779.slice/crio-e55ab73f793a03ed1c4e008d026d509100c25036d0e0952a313282299dfaa356 WatchSource:0}: Error finding container e55ab73f793a03ed1c4e008d026d509100c25036d0e0952a313282299dfaa356: Status 404 returned error can't find the container with id e55ab73f793a03ed1c4e008d026d509100c25036d0e0952a313282299dfaa356 Jul 08 13:31:03.570642 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:03.570613 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:31:03.572801 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:31:03.572771 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f49e71_c028_4bb5_ac4a_852a80d7c318.slice/crio-820e64320cb932584737525d3787cb9eead26d923a27363e167a2444e06cb3b3 WatchSource:0}: Error finding container 820e64320cb932584737525d3787cb9eead26d923a27363e167a2444e06cb3b3: Status 404 returned error can't find the container with id 820e64320cb932584737525d3787cb9eead26d923a27363e167a2444e06cb3b3 Jul 08 13:31:04.528261 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:04.528224 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-v88p8" event={"ID":"e8f49e71-c028-4bb5-ac4a-852a80d7c318","Type":"ContainerStarted","Data":"820e64320cb932584737525d3787cb9eead26d923a27363e167a2444e06cb3b3"} Jul 08 13:31:04.529878 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:04.529843 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" event={"ID":"c86d6fa1-eed5-4586-a6eb-9ab7dda80779","Type":"ContainerStarted","Data":"e55ab73f793a03ed1c4e008d026d509100c25036d0e0952a313282299dfaa356"} Jul 08 13:31:07.549449 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.549401 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-v88p8" event={"ID":"e8f49e71-c028-4bb5-ac4a-852a80d7c318","Type":"ContainerStarted","Data":"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416"} Jul 08 13:31:07.549906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.549550 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:07.550926 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.550901 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" event={"ID":"c86d6fa1-eed5-4586-a6eb-9ab7dda80779","Type":"ContainerStarted","Data":"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48"} Jul 08 13:31:07.551026 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.550985 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:07.568805 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.568757 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d5d4c5795-v88p8" podStartSLOduration=1.6565165830000002 podStartE2EDuration="4.568742539s" totalImagesPullingTime="2.912225956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:31:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:31:07.566794119 +0000 UTC m=+506.060513931" watchObservedRunningTime="2026-07-08 13:31:07.568742539 +0000 UTC m=+506.062462328" Jul 08 13:31:07.585483 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:07.585416 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" podStartSLOduration=2.625424798 podStartE2EDuration="5.585395915s" totalImagesPullingTime="2.959971117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:31:07.581200997 +0000 UTC m=+506.074920824" watchObservedRunningTime="2026-07-08 13:31:07.585395915 +0000 UTC m=+506.079115704" Jul 08 13:31:18.561076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.561047 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:18.561514 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.561115 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:31:18.623395 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.623356 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:18.623614 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.623587 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" podUID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" containerName="manager" containerID="cri-o://ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48" gracePeriod=30 Jul 08 13:31:18.872609 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.872583 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:18.963478 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.963444 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert\") pod \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " Jul 08 13:31:18.963658 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.963514 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw8l7\" (UniqueName: \"kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7\") pod \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\" (UID: \"c86d6fa1-eed5-4586-a6eb-9ab7dda80779\") " Jul 08 13:31:18.965694 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.965653 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "c86d6fa1-eed5-4586-a6eb-9ab7dda80779" (UID: "c86d6fa1-eed5-4586-a6eb-9ab7dda80779"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:31:18.965804 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:18.965699 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7" (OuterVolumeSpecName: "kube-api-access-jw8l7") pod "c86d6fa1-eed5-4586-a6eb-9ab7dda80779" (UID: "c86d6fa1-eed5-4586-a6eb-9ab7dda80779"). InnerVolumeSpecName "kube-api-access-jw8l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:31:19.064248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.064214 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-webhook-cert\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:31:19.064248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.064242 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8l7\" (UniqueName: \"kubernetes.io/projected/c86d6fa1-eed5-4586-a6eb-9ab7dda80779-kube-api-access-jw8l7\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:31:19.610824 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.610787 2576 generic.go:356] "Generic (PLEG): container finished" podID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" containerID="ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48" exitCode=0 Jul 08 13:31:19.611244 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.610843 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" Jul 08 13:31:19.611244 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.610872 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" event={"ID":"c86d6fa1-eed5-4586-a6eb-9ab7dda80779","Type":"ContainerDied","Data":"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48"} Jul 08 13:31:19.611244 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.610944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7d7b4dd59-w5t9x" event={"ID":"c86d6fa1-eed5-4586-a6eb-9ab7dda80779","Type":"ContainerDied","Data":"e55ab73f793a03ed1c4e008d026d509100c25036d0e0952a313282299dfaa356"} Jul 08 13:31:19.611244 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.610966 2576 scope.go:117] "RemoveContainer" containerID="ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48" Jul 08 13:31:19.619736 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.619722 2576 scope.go:117] "RemoveContainer" containerID="ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48" Jul 08 13:31:19.619966 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:31:19.619950 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48\": container with ID starting with ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48 not found: ID does not exist" containerID="ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48" Jul 08 13:31:19.620009 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.619972 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48"} err="failed to get container status \"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48\": rpc error: code = NotFound desc = could not find container \"ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48\": container with ID starting with ce5dc5185610432da6cf32416d94588d653e17562cee845f050d9c0402b02f48 not found: ID does not exist" Jul 08 13:31:19.632192 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.632156 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:19.635676 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:19.635655 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7d7b4dd59-w5t9x"] Jul 08 13:31:20.043948 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:20.043914 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" path="/var/lib/kubelet/pods/c86d6fa1-eed5-4586-a6eb-9ab7dda80779/volumes" Jul 08 13:31:39.629093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.629015 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd"] Jul 08 13:31:39.629565 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.629433 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" containerName="manager" Jul 08 13:31:39.629565 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.629446 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" containerName="manager" Jul 08 13:31:39.629565 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.629528 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c86d6fa1-eed5-4586-a6eb-9ab7dda80779" containerName="manager" Jul 08 13:31:39.631734 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.631713 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.636211 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.636162 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hz4c8\"" Jul 08 13:31:39.636354 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.636209 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 13:31:39.651553 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.651524 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd"] Jul 08 13:31:39.737462 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.737430 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f62a67-c30e-4d86-93fb-842417db12a6-maas-api-tls\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.737618 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.737499 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f62a67-c30e-4d86-93fb-842417db12a6-openshift-service-ca\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.737618 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.737522 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rjlb\" (UniqueName: \"kubernetes.io/projected/d3f62a67-c30e-4d86-93fb-842417db12a6-kube-api-access-7rjlb\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.838695 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.838660 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f62a67-c30e-4d86-93fb-842417db12a6-openshift-service-ca\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.838695 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.838697 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rjlb\" (UniqueName: \"kubernetes.io/projected/d3f62a67-c30e-4d86-93fb-842417db12a6-kube-api-access-7rjlb\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.838877 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.838776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f62a67-c30e-4d86-93fb-842417db12a6-maas-api-tls\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.839454 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.839428 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3f62a67-c30e-4d86-93fb-842417db12a6-openshift-service-ca\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.841187 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.841151 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3f62a67-c30e-4d86-93fb-842417db12a6-maas-api-tls\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.847971 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.847945 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rjlb\" (UniqueName: \"kubernetes.io/projected/d3f62a67-c30e-4d86-93fb-842417db12a6-kube-api-access-7rjlb\") pod \"maas-api-fb96799d5-w8jgd\" (UID: \"d3f62a67-c30e-4d86-93fb-842417db12a6\") " pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:39.942511 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:39.942476 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:40.101834 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.101799 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-p97vl"] Jul 08 13:31:40.116105 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.116069 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-p97vl"] Jul 08 13:31:40.116253 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.116200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.117670 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:31:40.117638 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3f62a67_c30e_4d86_93fb_842417db12a6.slice/crio-67d7b47a40e5e2c605ab0ac89eded369d61c12f37f5fb78391a68719a322b0af WatchSource:0}: Error finding container 67d7b47a40e5e2c605ab0ac89eded369d61c12f37f5fb78391a68719a322b0af: Status 404 returned error can't find the container with id 67d7b47a40e5e2c605ab0ac89eded369d61c12f37f5fb78391a68719a322b0af Jul 08 13:31:40.117785 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.117713 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd"] Jul 08 13:31:40.119045 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.119024 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 13:31:40.119045 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.119036 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jvmxj\"" Jul 08 13:31:40.130448 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.130425 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fs5q7"] Jul 08 13:31:40.140884 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.140865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.145302 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.145276 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fs5q7"] Jul 08 13:31:40.241599 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.241518 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmkph\" (UniqueName: \"kubernetes.io/projected/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-kube-api-access-kmkph\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.241599 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.241571 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rjl\" (UniqueName: \"kubernetes.io/projected/94864546-69fa-402d-8611-c214e1166ed1-kube-api-access-t2rjl\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.241822 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.241685 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-plugins-config-volume\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.241822 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.241717 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94864546-69fa-402d-8611-c214e1166ed1-plugins-config-volume\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.343070 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343037 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-plugins-config-volume\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.343070 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94864546-69fa-402d-8611-c214e1166ed1-plugins-config-volume\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.343315 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343123 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmkph\" (UniqueName: \"kubernetes.io/projected/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-kube-api-access-kmkph\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.343315 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343164 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rjl\" (UniqueName: \"kubernetes.io/projected/94864546-69fa-402d-8611-c214e1166ed1-kube-api-access-t2rjl\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.343791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343770 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-plugins-config-volume\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.343864 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.343845 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94864546-69fa-402d-8611-c214e1166ed1-plugins-config-volume\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.351878 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.351855 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rjl\" (UniqueName: \"kubernetes.io/projected/94864546-69fa-402d-8611-c214e1166ed1-kube-api-access-t2rjl\") pod \"payload-processing-58f86c7798-fs5q7\" (UID: \"94864546-69fa-402d-8611-c214e1166ed1\") " pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.351998 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.351886 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmkph\" (UniqueName: \"kubernetes.io/projected/44dcf8eb-4c81-48aa-88ce-7ea8b217e018-kube-api-access-kmkph\") pod \"payload-pre-processing-7964457696-p97vl\" (UID: \"44dcf8eb-4c81-48aa-88ce-7ea8b217e018\") " pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.436570 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.436528 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:40.451430 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.451396 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:40.605920 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.605885 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-p97vl"] Jul 08 13:31:40.610380 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:31:40.610322 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44dcf8eb_4c81_48aa_88ce_7ea8b217e018.slice/crio-7272eda649c86efea91b56dbb60274005c9538b2557f5f3cf3840b4d61e1b7dd WatchSource:0}: Error finding container 7272eda649c86efea91b56dbb60274005c9538b2557f5f3cf3840b4d61e1b7dd: Status 404 returned error can't find the container with id 7272eda649c86efea91b56dbb60274005c9538b2557f5f3cf3840b4d61e1b7dd Jul 08 13:31:40.644410 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.644357 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-fs5q7"] Jul 08 13:31:40.720518 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.720475 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" event={"ID":"94864546-69fa-402d-8611-c214e1166ed1","Type":"ContainerStarted","Data":"f9508068b6fa215f91dc956132f41b65a2fa233100d31e4c9251f3af91724483"} Jul 08 13:31:40.721642 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.721613 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" event={"ID":"44dcf8eb-4c81-48aa-88ce-7ea8b217e018","Type":"ContainerStarted","Data":"7272eda649c86efea91b56dbb60274005c9538b2557f5f3cf3840b4d61e1b7dd"} Jul 08 13:31:40.722791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:40.722756 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" event={"ID":"d3f62a67-c30e-4d86-93fb-842417db12a6","Type":"ContainerStarted","Data":"67d7b47a40e5e2c605ab0ac89eded369d61c12f37f5fb78391a68719a322b0af"} Jul 08 13:31:42.738501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.738455 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" event={"ID":"44dcf8eb-4c81-48aa-88ce-7ea8b217e018","Type":"ContainerStarted","Data":"b9ce80c6292aa6dd4ef7573cdf165c329ded2c3f184e8ff0dbb556615fa99c6f"} Jul 08 13:31:42.738848 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.738600 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:42.740358 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.740332 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" event={"ID":"d3f62a67-c30e-4d86-93fb-842417db12a6","Type":"ContainerStarted","Data":"00169181a04a7656a89ce5578f51b35779951a5f28f9555434b8982c12371e65"} Jul 08 13:31:42.740471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.740421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:42.741835 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.741814 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" event={"ID":"94864546-69fa-402d-8611-c214e1166ed1","Type":"ContainerStarted","Data":"77bb15aa196f881d49fe68204c6f578b941cddaccd82088e191d6c53b56e7026"} Jul 08 13:31:42.742280 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.742253 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:31:42.756223 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.755490 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" podStartSLOduration=0.806142436 podStartE2EDuration="2.75547326s" totalImagesPullingTime="1.949330824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:31:42.753945792 +0000 UTC m=+541.247665581" watchObservedRunningTime="2026-07-08 13:31:42.75547326 +0000 UTC m=+541.249193050" Jul 08 13:31:42.790231 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:42.790152 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" podStartSLOduration=0.878150575 podStartE2EDuration="2.790128854s" totalImagesPullingTime="1.911978279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:31:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:31:42.773930898 +0000 UTC m=+541.267650691" watchObservedRunningTime="2026-07-08 13:31:42.790128854 +0000 UTC m=+541.283848645" Jul 08 13:31:47.768568 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.768536 2576 generic.go:356] "Generic (PLEG): container finished" podID="30918a42-4d90-4bff-83ce-b2facafdd2b8" containerID="9352a322a813e0a529661a617cff8b7f1ee817a0e66230f8161f8236336464d1" exitCode=0 Jul 08 13:31:47.769037 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.768608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" event={"ID":"30918a42-4d90-4bff-83ce-b2facafdd2b8","Type":"ContainerDied","Data":"9352a322a813e0a529661a617cff8b7f1ee817a0e66230f8161f8236336464d1"} Jul 08 13:31:47.769037 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.769015 2576 scope.go:117] "RemoveContainer" containerID="9352a322a813e0a529661a617cff8b7f1ee817a0e66230f8161f8236336464d1" Jul 08 13:31:47.769959 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.769930 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:31:47.770024 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.769994 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:31:47.770069 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.770024 2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 08 13:31:47.791280 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:47.791226 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" podStartSLOduration=6.3480662 podStartE2EDuration="8.791208595s" totalImagesPullingTime="2.443142395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:31:42.792061776 +0000 UTC m=+541.285781582" watchObservedRunningTime="2026-07-08 13:31:47.791208595 +0000 UTC m=+546.284928386" Jul 08 13:31:48.752691 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:48.752659 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-fb96799d5-w8jgd" Jul 08 13:31:48.775514 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:48.775484 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" event={"ID":"30918a42-4d90-4bff-83ce-b2facafdd2b8","Type":"ContainerStarted","Data":"e209d8c10254fe6ee5f74098a5b3d349522ee324ff5da51adf2970014c7a7e81"} Jul 08 13:31:49.754740 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:49.754705 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:31:49.754917 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:49.754896 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fn5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 13:31:49.754966 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:49.754943 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" podUID="30918a42-4d90-4bff-83ce-b2facafdd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 13:31:50.754276 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:50.754237 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fn5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 13:31:50.754658 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:50.754297 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" podUID="30918a42-4d90-4bff-83ce-b2facafdd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 13:31:51.754943 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:51.754898 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fn5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 13:31:51.755442 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:51.754951 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" podUID="30918a42-4d90-4bff-83ce-b2facafdd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 13:31:52.755102 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:52.755065 2576 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-fn5sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 13:31:52.755583 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:52.755139 2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" podUID="30918a42-4d90-4bff-83ce-b2facafdd2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 13:31:53.749384 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:53.749350 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-p97vl" Jul 08 13:31:53.757371 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:53.757346 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:31:53.757791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:53.757692 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:31:53.758283 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:53.758266 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-fn5sw" Jul 08 13:31:54.754088 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:31:54.754059 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-fs5q7" Jul 08 13:32:04.984446 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.984412 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj"] Jul 08 13:32:04.987411 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.987390 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:04.989883 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.989859 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2p2fd\"" Jul 08 13:32:04.990227 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.990211 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 13:32:04.990307 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.990263 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 13:32:04.990972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.990958 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 08 13:32:04.998870 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:04.998845 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj"] Jul 08 13:32:05.003189 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.003146 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43a11e4-a795-4354-928a-da90ad4d1f56-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.003515 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.003489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.003702 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.003679 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.003892 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.003875 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.004048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.004030 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24fm\" (UniqueName: \"kubernetes.io/projected/b43a11e4-a795-4354-928a-da90ad4d1f56-kube-api-access-x24fm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.004278 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.004254 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.104784 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104749 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.104784 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104791 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24fm\" (UniqueName: \"kubernetes.io/projected/b43a11e4-a795-4354-928a-da90ad4d1f56-kube-api-access-x24fm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43a11e4-a795-4354-928a-da90ad4d1f56-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.104975 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105297 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.105271 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105381 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.105343 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.105474 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.105456 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.107224 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.107201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b43a11e4-a795-4354-928a-da90ad4d1f56-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.107557 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.107538 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b43a11e4-a795-4354-928a-da90ad4d1f56-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.113031 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.113003 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24fm\" (UniqueName: \"kubernetes.io/projected/b43a11e4-a795-4354-928a-da90ad4d1f56-kube-api-access-x24fm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bkjbj\" (UID: \"b43a11e4-a795-4354-928a-da90ad4d1f56\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.298329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.298227 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:05.432543 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.432519 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj"] Jul 08 13:32:05.434585 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:32:05.434552 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43a11e4_a795_4354_928a_da90ad4d1f56.slice/crio-6b7d71fb9555915b52a0b468b9c3621b5500705fb977aa43513c767d1644ed18 WatchSource:0}: Error finding container 6b7d71fb9555915b52a0b468b9c3621b5500705fb977aa43513c767d1644ed18: Status 404 returned error can't find the container with id 6b7d71fb9555915b52a0b468b9c3621b5500705fb977aa43513c767d1644ed18 Jul 08 13:32:05.872542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:05.872504 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" event={"ID":"b43a11e4-a795-4354-928a-da90ad4d1f56","Type":"ContainerStarted","Data":"6b7d71fb9555915b52a0b468b9c3621b5500705fb977aa43513c767d1644ed18"} Jul 08 13:32:08.891939 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:08.891902 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" event={"ID":"b43a11e4-a795-4354-928a-da90ad4d1f56","Type":"ContainerStarted","Data":"3b99f1cc5cd9cba127c86a1859f5936efacefce2a7ee6c2ad1ca527fe8fc1c75"} Jul 08 13:32:08.909664 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:08.909615 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" podStartSLOduration=2.132614887 podStartE2EDuration="4.909600455s" totalImagesPullingTime="2.776985568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:32:08.909076125 +0000 UTC m=+567.402795913" watchObservedRunningTime="2026-07-08 13:32:08.909600455 +0000 UTC m=+567.403320240" Jul 08 13:32:15.299332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.299299 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:15.299894 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.299358 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:15.311664 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.311640 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:15.397709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.397669 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t"] Jul 08 13:32:15.405105 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.405062 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.409728 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.409678 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 13:32:15.413082 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.413055 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t"] Jul 08 13:32:15.499384 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.499569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdhm5\" (UniqueName: \"kubernetes.io/projected/bd0f3a20-d752-4cc0-9571-78b559e31669-kube-api-access-mdhm5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.499569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499464 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.499569 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f3a20-d752-4cc0-9571-78b559e31669-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.499727 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499603 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.499727 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.499629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.600849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f3a20-d752-4cc0-9571-78b559e31669-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.600849 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601038 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601038 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601038 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600965 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdhm5\" (UniqueName: \"kubernetes.io/projected/bd0f3a20-d752-4cc0-9571-78b559e31669-kube-api-access-mdhm5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601038 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.600998 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601393 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.601349 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601530 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.601423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.601530 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.601465 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.603284 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.603264 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bd0f3a20-d752-4cc0-9571-78b559e31669-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.603440 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.603423 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bd0f3a20-d752-4cc0-9571-78b559e31669-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.608745 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.608724 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdhm5\" (UniqueName: \"kubernetes.io/projected/bd0f3a20-d752-4cc0-9571-78b559e31669-kube-api-access-mdhm5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t\" (UID: \"bd0f3a20-d752-4cc0-9571-78b559e31669\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.720915 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.720874 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:15.855972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.855946 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t"] Jul 08 13:32:15.857809 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:32:15.857770 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd0f3a20_d752_4cc0_9571_78b559e31669.slice/crio-43b9316a0b6b32b2c2be15cc6d5228d19d7e46f92d82c4ded90e97c0d55ed2df WatchSource:0}: Error finding container 43b9316a0b6b32b2c2be15cc6d5228d19d7e46f92d82c4ded90e97c0d55ed2df: Status 404 returned error can't find the container with id 43b9316a0b6b32b2c2be15cc6d5228d19d7e46f92d82c4ded90e97c0d55ed2df Jul 08 13:32:15.936161 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.936122 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" event={"ID":"bd0f3a20-d752-4cc0-9571-78b559e31669","Type":"ContainerStarted","Data":"43b9316a0b6b32b2c2be15cc6d5228d19d7e46f92d82c4ded90e97c0d55ed2df"} Jul 08 13:32:15.947445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:15.947421 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bkjbj" Jul 08 13:32:16.942979 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:16.942944 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" event={"ID":"bd0f3a20-d752-4cc0-9571-78b559e31669","Type":"ContainerStarted","Data":"e62fa68a26d3d2a4b34859e39f8992db219cdcb9f7d750ec8e6e55580a834123"} Jul 08 13:32:16.960587 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:16.960531 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" podStartSLOduration=1.047893937 podStartE2EDuration="1.960514914s" totalImagesPullingTime="912.620977ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:32:16.959660845 +0000 UTC m=+575.453380637" watchObservedRunningTime="2026-07-08 13:32:16.960514914 +0000 UTC m=+575.454234704" Jul 08 13:32:25.721952 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:25.721910 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:25.722355 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:25.721979 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:25.734473 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:25.734445 2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:26.001136 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:26.001053 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t" Jul 08 13:32:41.972611 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:41.972571 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:32:41.973800 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:32:41.973780 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:33:59.233810 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.233724 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-69899986ff-49d8n"] Jul 08 13:33:59.237556 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.237531 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.245675 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.245647 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69899986ff-49d8n"] Jul 08 13:33:59.323233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.323197 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-tls-cert\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.323402 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.323294 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcx9\" (UniqueName: \"kubernetes.io/projected/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-kube-api-access-qmcx9\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.424485 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.424449 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcx9\" (UniqueName: \"kubernetes.io/projected/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-kube-api-access-qmcx9\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.424485 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.424488 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-tls-cert\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.426893 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.426874 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-tls-cert\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.432077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.432052 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcx9\" (UniqueName: \"kubernetes.io/projected/c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e-kube-api-access-qmcx9\") pod \"authorino-69899986ff-49d8n\" (UID: \"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e\") " pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.548024 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.547927 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-69899986ff-49d8n" Jul 08 13:33:59.686410 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.684715 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-69899986ff-49d8n"] Jul 08 13:33:59.689645 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:33:59.689618 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc4fb8b4d_b974_4fd0_b80c_4b5eec62f62e.slice/crio-3c52792d2eb8b570fb7cbd8e7af94fccff17123c6bead00bb7587f6b3d3cbb1c WatchSource:0}: Error finding container 3c52792d2eb8b570fb7cbd8e7af94fccff17123c6bead00bb7587f6b3d3cbb1c: Status 404 returned error can't find the container with id 3c52792d2eb8b570fb7cbd8e7af94fccff17123c6bead00bb7587f6b3d3cbb1c Jul 08 13:33:59.691250 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:33:59.691223 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:34:00.478263 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.478221 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69899986ff-49d8n" event={"ID":"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e","Type":"ContainerStarted","Data":"3d4e7723f795db4dd8447882d5835443334cb74b3f4b90e94a40260bc64ae840"} Jul 08 13:34:00.478746 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.478279 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-69899986ff-49d8n" event={"ID":"c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e","Type":"ContainerStarted","Data":"3c52792d2eb8b570fb7cbd8e7af94fccff17123c6bead00bb7587f6b3d3cbb1c"} Jul 08 13:34:00.497252 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.497195 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-69899986ff-49d8n" podStartSLOduration=1.064356225 podStartE2EDuration="1.497161244s" totalImagesPullingTime="432.805019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:34:00.49484858 +0000 UTC m=+678.988568370" watchObservedRunningTime="2026-07-08 13:34:00.497161244 +0000 UTC m=+678.990881106" Jul 08 13:34:00.527217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.526157 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:34:00.527217 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.527129 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-86f8d7798b-x68jj" podUID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" containerName="authorino" containerID="cri-o://35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c" gracePeriod=30 Jul 08 13:34:00.793377 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.793352 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:34:00.847903 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.847871 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msjcl\" (UniqueName: \"kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl\") pod \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " Jul 08 13:34:00.848112 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.847931 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert\") pod \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\" (UID: \"63ff4ef1-f14a-403f-92b7-6cff0a24efb0\") " Jul 08 13:34:00.850660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.850613 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl" (OuterVolumeSpecName: "kube-api-access-msjcl") pod "63ff4ef1-f14a-403f-92b7-6cff0a24efb0" (UID: "63ff4ef1-f14a-403f-92b7-6cff0a24efb0"). InnerVolumeSpecName "kube-api-access-msjcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:34:00.861193 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.861145 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "63ff4ef1-f14a-403f-92b7-6cff0a24efb0" (UID: "63ff4ef1-f14a-403f-92b7-6cff0a24efb0"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:34:00.949682 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.949631 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msjcl\" (UniqueName: \"kubernetes.io/projected/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-kube-api-access-msjcl\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:34:00.949682 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:00.949660 2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/63ff4ef1-f14a-403f-92b7-6cff0a24efb0-tls-cert\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:34:01.484790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.484754 2576 generic.go:356] "Generic (PLEG): container finished" podID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" containerID="35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c" exitCode=0 Jul 08 13:34:01.484790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.484795 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-86f8d7798b-x68jj" Jul 08 13:34:01.485416 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.484836 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86f8d7798b-x68jj" event={"ID":"63ff4ef1-f14a-403f-92b7-6cff0a24efb0","Type":"ContainerDied","Data":"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c"} Jul 08 13:34:01.485416 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.484886 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-86f8d7798b-x68jj" event={"ID":"63ff4ef1-f14a-403f-92b7-6cff0a24efb0","Type":"ContainerDied","Data":"30ba6735c57484a8edc9a01f40f3efd6ce34c9ad65f0b4f196ac9f3e13f56482"} Jul 08 13:34:01.485416 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.484906 2576 scope.go:117] "RemoveContainer" containerID="35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c" Jul 08 13:34:01.496489 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.496467 2576 scope.go:117] "RemoveContainer" containerID="35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c" Jul 08 13:34:01.496785 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:34:01.496767 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c\": container with ID starting with 35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c not found: ID does not exist" containerID="35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c" Jul 08 13:34:01.496853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.496792 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c"} err="failed to get container status \"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c\": rpc error: code = NotFound desc = could not find container \"35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c\": container with ID starting with 35fca26817252161bf3d2fdf3b76035ef2b6abebac4f2aa4540f782edef9b51c not found: ID does not exist" Jul 08 13:34:01.509385 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.509355 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:34:01.513541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:01.513513 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-86f8d7798b-x68jj"] Jul 08 13:34:02.045735 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:34:02.045695 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" path="/var/lib/kubelet/pods/63ff4ef1-f14a-403f-92b7-6cff0a24efb0/volumes" Jul 08 13:35:33.222050 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.222009 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:35:33.222570 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.222283 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-d5d4c5795-v88p8" podUID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" containerName="manager" containerID="cri-o://3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416" gracePeriod=30 Jul 08 13:35:33.470146 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.470118 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:35:33.495196 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.495088 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9wvd\" (UniqueName: \"kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd\") pod \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " Jul 08 13:35:33.495196 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.495159 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert\") pod \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\" (UID: \"e8f49e71-c028-4bb5-ac4a-852a80d7c318\") " Jul 08 13:35:33.497290 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.497256 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd" (OuterVolumeSpecName: "kube-api-access-r9wvd") pod "e8f49e71-c028-4bb5-ac4a-852a80d7c318" (UID: "e8f49e71-c028-4bb5-ac4a-852a80d7c318"). InnerVolumeSpecName "kube-api-access-r9wvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:35:33.498109 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.498077 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e8f49e71-c028-4bb5-ac4a-852a80d7c318" (UID: "e8f49e71-c028-4bb5-ac4a-852a80d7c318"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:35:33.596510 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.596477 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9wvd\" (UniqueName: \"kubernetes.io/projected/e8f49e71-c028-4bb5-ac4a-852a80d7c318-kube-api-access-r9wvd\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:35:33.596510 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.596507 2576 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e8f49e71-c028-4bb5-ac4a-852a80d7c318-webhook-cert\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:35:33.957980 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.957950 2576 generic.go:356] "Generic (PLEG): container finished" podID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" containerID="3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416" exitCode=0 Jul 08 13:35:33.958156 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.957993 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-v88p8" Jul 08 13:35:33.958156 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.958037 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-v88p8" event={"ID":"e8f49e71-c028-4bb5-ac4a-852a80d7c318","Type":"ContainerDied","Data":"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416"} Jul 08 13:35:33.958156 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.958092 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-v88p8" event={"ID":"e8f49e71-c028-4bb5-ac4a-852a80d7c318","Type":"ContainerDied","Data":"820e64320cb932584737525d3787cb9eead26d923a27363e167a2444e06cb3b3"} Jul 08 13:35:33.958156 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.958117 2576 scope.go:117] "RemoveContainer" containerID="3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416" Jul 08 13:35:33.967718 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.967597 2576 scope.go:117] "RemoveContainer" containerID="3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416" Jul 08 13:35:33.967870 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:35:33.967853 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416\": container with ID starting with 3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416 not found: ID does not exist" containerID="3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416" Jul 08 13:35:33.967912 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.967877 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416"} err="failed to get container status \"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416\": rpc error: code = NotFound desc = could not find container \"3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416\": container with ID starting with 3bfb8d025acb1a2a125b4a21c04dd71956f42d1906792124f5bf937fd4f20416 not found: ID does not exist" Jul 08 13:35:33.979732 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.979706 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:35:33.983598 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:33.983571 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-v88p8"] Jul 08 13:35:34.044666 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.044632 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" path="/var/lib/kubelet/pods/e8f49e71-c028-4bb5-ac4a-852a80d7c318/volumes" Jul 08 13:35:34.780449 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780416 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d5d4c5795-gvv6w"] Jul 08 13:35:34.780856 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780842 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" containerName="manager" Jul 08 13:35:34.780856 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780855 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" containerName="manager" Jul 08 13:35:34.780940 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780883 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" containerName="authorino" Jul 08 13:35:34.780940 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780889 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" containerName="authorino" Jul 08 13:35:34.781016 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780942 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="63ff4ef1-f14a-403f-92b7-6cff0a24efb0" containerName="authorino" Jul 08 13:35:34.781016 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.780952 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8f49e71-c028-4bb5-ac4a-852a80d7c318" containerName="manager" Jul 08 13:35:34.785362 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.785345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:34.787801 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.787782 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 13:35:34.787899 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.787782 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gxfhm\"" Jul 08 13:35:34.788730 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.788709 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 13:35:34.791813 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.791787 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-gvv6w"] Jul 08 13:35:34.908853 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.908807 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734d70ba-2a07-4e45-ab64-289441a400ec-webhook-cert\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:34.909043 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:34.908896 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc7ln\" (UniqueName: \"kubernetes.io/projected/734d70ba-2a07-4e45-ab64-289441a400ec-kube-api-access-wc7ln\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.010271 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.010235 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734d70ba-2a07-4e45-ab64-289441a400ec-webhook-cert\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.010476 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.010305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc7ln\" (UniqueName: \"kubernetes.io/projected/734d70ba-2a07-4e45-ab64-289441a400ec-kube-api-access-wc7ln\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.012712 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.012680 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/734d70ba-2a07-4e45-ab64-289441a400ec-webhook-cert\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.022160 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.022135 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc7ln\" (UniqueName: \"kubernetes.io/projected/734d70ba-2a07-4e45-ab64-289441a400ec-kube-api-access-wc7ln\") pod \"maas-controller-d5d4c5795-gvv6w\" (UID: \"734d70ba-2a07-4e45-ab64-289441a400ec\") " pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.097115 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.097020 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:35.233341 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.233314 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d5d4c5795-gvv6w"] Jul 08 13:35:35.235944 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:35:35.235915 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734d70ba_2a07_4e45_ab64_289441a400ec.slice/crio-a034480545f9d1fa09103d9c7a524734819adcac43598bb37ce43f99d306c609 WatchSource:0}: Error finding container a034480545f9d1fa09103d9c7a524734819adcac43598bb37ce43f99d306c609: Status 404 returned error can't find the container with id a034480545f9d1fa09103d9c7a524734819adcac43598bb37ce43f99d306c609 Jul 08 13:35:35.971379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:35.971343 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" event={"ID":"734d70ba-2a07-4e45-ab64-289441a400ec","Type":"ContainerStarted","Data":"a034480545f9d1fa09103d9c7a524734819adcac43598bb37ce43f99d306c609"} Jul 08 13:35:36.977645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:36.977606 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" event={"ID":"734d70ba-2a07-4e45-ab64-289441a400ec","Type":"ContainerStarted","Data":"d53de2b71033467ec8c79fc246c397abd7b65af97df9e721b3d8bed556d70945"} Jul 08 13:35:36.978051 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:36.977676 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:35:36.995574 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:36.995525 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" podStartSLOduration=2.166507723 podStartE2EDuration="2.995511717s" totalImagesPullingTime="829.003994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:35:36.994614725 +0000 UTC m=+775.488334546" watchObservedRunningTime="2026-07-08 13:35:36.995511717 +0000 UTC m=+775.489231500" Jul 08 13:35:47.988472 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:35:47.988440 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d5d4c5795-gvv6w" Jul 08 13:36:42.592536 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:36:42.592506 2576 scope.go:117] "RemoveContainer" containerID="6361a97f5c4e06978a6466255b1700dcf177ac674803a70374ee1b91a08a5865" Jul 08 13:37:20.805685 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:37:20.805628 2576 controller.go:195] "Failed to update lease" err="Put \"https://affb78f60cad14e6fb0028e1ef483bdc-f103e07a3a87b1d0.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-212.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 08 13:37:30.806451 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:37:30.806405 2576 controller.go:195] "Failed to update lease" err="Put \"https://affb78f60cad14e6fb0028e1ef483bdc-f103e07a3a87b1d0.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-212.ec2.internal?timeout=10s\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Jul 08 13:37:40.807321 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:37:40.807225 2576 controller.go:195] "Failed to update lease" err="Put \"https://affb78f60cad14e6fb0028e1ef483bdc-f103e07a3a87b1d0.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-212.ec2.internal?timeout=10s\": context deadline exceeded" Jul 08 13:37:42.008070 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:42.008040 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:37:42.008476 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:42.008147 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:37:49.874121 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874081 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874121 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874101 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874121 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874070 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874121 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874073 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874132 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874154 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874161 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874215 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874227 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874221 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-2p2fd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874237 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zxlzq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874247 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9qkf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874268 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874282 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874290 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874299 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874307 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z4knk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874318 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874355 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874357 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874381 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.874764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874408 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hz4c8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874425 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874431 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874449 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q9lvl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874457 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874466 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874514 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874522 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874533 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874518 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874561 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874565 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874580 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874591 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874598 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874612 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874631 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xfwk6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874653 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.875630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874686 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874688 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874708 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874712 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874724 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874732 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jbp9x\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874756 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874774 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874779 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874806 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7bmpq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874821 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874825 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874827 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pfpsg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874844 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874848 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7qjwb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874860 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jvmxj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874872 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874891 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-dockercfg-rxggx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876334 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874910 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874897 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874928 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cwv99\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874955 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874958 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-njqq9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874960 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874972 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gxfhm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.874987 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875001 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875004 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875020 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875028 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vbgjp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875030 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875032 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875052 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"default-dockercfg-5xcqm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875071 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h4rms\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875080 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"maas-parameters\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875105 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.876901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875129 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875145 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875160 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875166 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875162 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875198 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875146 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875214 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6j9wp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875229 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875232 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875247 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875261 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875276 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875295 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5pd96\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875203 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875302 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875310 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875278 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875329 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875345 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875344 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875235 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875364 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875373 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dfn6w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875383 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875264 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxf8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875395 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875403 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875413 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrr5t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875426 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875428 2576 reflector.go:571] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875438 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875443 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875380 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-76cv9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875461 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875408 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.877989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875472 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875486 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6blql\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875502 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875345 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875514 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875476 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875530 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zb85s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875551 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875569 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875574 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875553 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rlw5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875587 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875595 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"signing-key\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875595 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hjkcl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875510 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875617 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8mtsb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875633 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875639 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.878542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875529 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875636 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tktlz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875676 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l8qx6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875680 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875684 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zs278\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875687 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875693 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"default-dockercfg-prcmd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875610 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875707 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875710 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpm48\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875572 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kj4bg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875710 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875732 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qldzq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875736 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lc5hv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875744 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875739 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875739 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875760 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879076 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875712 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvhmk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875708 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875776 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875778 2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875786 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q5z89\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875793 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875797 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zxl8v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875722 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875773 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875815 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875760 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875739 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-72q2p\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:37:49.875834 2576 controller.go:195] "Failed to update lease" err="Put \"https://affb78f60cad14e6fb0028e1ef483bdc-f103e07a3a87b1d0.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-138-212.ec2.internal?timeout=10s\": http2: client connection lost" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875799 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hs9z4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875805 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:49.879622 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:49.875811 2576 reflector.go:571] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding" Jul 08 13:37:50.679445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.679414 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 13:37:50.696103 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.696076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-7bmpq\"" Jul 08 13:37:50.698990 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.698968 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x9qkf\"" Jul 08 13:37:50.699781 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.699764 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-q9lvl\"" Jul 08 13:37:50.704425 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.704407 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.726341 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.726315 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 13:37:50.727072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.727057 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 13:37:50.728014 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.727999 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 13:37:50.739648 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.739622 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.751935 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.751906 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.753022 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.753006 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 08 13:37:50.759331 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.759311 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 13:37:50.760434 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.760418 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zs278\"" Jul 08 13:37:50.766782 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.766761 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 13:37:50.776183 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.776154 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 13:37:50.777781 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.777766 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-zxlzq\"" Jul 08 13:37:50.779312 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.779296 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 13:37:50.782343 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.782329 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-76cv9\"" Jul 08 13:37:50.790665 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.790643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 13:37:50.793506 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.793487 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.794959 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.794944 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.795201 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.795188 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 08 13:37:50.816843 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.816818 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 13:37:50.822419 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.822402 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cwv99\"" Jul 08 13:37:50.823736 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.823720 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-hz4c8\"" Jul 08 13:37:50.823736 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.823727 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.823860 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.823739 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.824091 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.824076 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-njqq9\"" Jul 08 13:37:50.829414 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.829401 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-zxl8v\"" Jul 08 13:37:50.836816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.836796 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-6j9wp\"" Jul 08 13:37:50.841243 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.841225 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hs9z4\"" Jul 08 13:37:50.842894 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.842871 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.849439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.849424 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.866138 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.866119 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 13:37:50.867921 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.867902 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fpm48\"" Jul 08 13:37:50.874117 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.874102 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 13:37:50.879487 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.879473 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.882531 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.882515 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-6blql\"" Jul 08 13:37:50.888126 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.888109 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-qldzq\"" Jul 08 13:37:50.891628 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.891607 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zb85s\"" Jul 08 13:37:50.895966 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.895948 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 13:37:50.901642 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.901625 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 13:37:50.902838 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.902824 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 13:37:50.908415 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.908402 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 08 13:37:50.911824 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.911808 2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:37:50.913905 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.913891 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 13:37:50.936275 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.936219 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.956004 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.955983 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2p2fd\"" Jul 08 13:37:50.981455 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.981435 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 13:37:50.987157 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.987142 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:50.997236 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.997218 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 13:37:50.997521 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:50.997507 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 13:37:51.002187 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.002158 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jvmxj\"" Jul 08 13:37:51.003542 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.003528 2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:37:51.006800 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.006786 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 13:37:51.011468 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.011454 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 13:37:51.023055 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.023037 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 13:37:51.026830 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.026815 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.027814 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.027801 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.031446 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.031430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l8qx6\"" Jul 08 13:37:51.041699 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.041682 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 13:37:51.042690 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.042676 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.052590 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.052561 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 13:37:51.052670 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.052561 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 13:37:51.052719 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.052688 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 13:37:51.063593 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.063574 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hjkcl\"" Jul 08 13:37:51.070300 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.070284 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 13:37:51.071309 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.071295 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 08 13:37:51.075291 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.075268 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 13:37:51.083562 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.083533 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.083674 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.083643 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lrr5t\"" Jul 08 13:37:51.087501 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.087478 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 13:37:51.094086 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.094068 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.100660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.100641 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.105155 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.105139 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 08 13:37:51.107906 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.107891 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tktlz\"" Jul 08 13:37:51.108498 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.108484 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 08 13:37:51.109260 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.109247 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.111620 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.111605 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 13:37:51.112452 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.112440 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5xcqm\"" Jul 08 13:37:51.114941 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.114926 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-gxfhm\"" Jul 08 13:37:51.131636 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.131616 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 13:37:51.138220 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.138198 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 08 13:37:51.144836 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.144816 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.145741 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.145729 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.150098 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.150083 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 13:37:51.153271 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.153257 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.154379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.154366 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 13:37:51.155924 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.155911 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxf8s\"" Jul 08 13:37:51.161313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.161298 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-72q2p\"" Jul 08 13:37:51.169680 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.169663 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.172865 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.172850 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-prcmd\"" Jul 08 13:37:51.174191 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.174162 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 13:37:51.177315 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.177301 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-q5z89\"" Jul 08 13:37:51.182673 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.182659 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.183787 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.183773 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.184727 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.184711 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 13:37:51.184769 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.184730 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 13:37:51.199587 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.199303 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kvhmk\"" Jul 08 13:37:51.201523 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.201505 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 13:37:51.202390 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.202371 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 13:37:51.205367 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.205350 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vbgjp\"" Jul 08 13:37:51.212013 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.211996 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.213034 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.213019 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.214840 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.214824 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.223825 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.223808 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.235631 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.235609 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-h4rms\"" Jul 08 13:37:51.256730 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.256707 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-jbp9x\"" Jul 08 13:37:51.260142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.260124 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 13:37:51.261033 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.261017 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 13:37:51.264445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.264429 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 08 13:37:51.265603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.265588 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.266447 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.266433 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9rlw5\"" Jul 08 13:37:51.267755 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.267737 2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 13:37:51.281458 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.281438 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 13:37:51.281698 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.281686 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 08 13:37:51.281738 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.281709 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 13:37:51.286200 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.286163 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.287376 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.287363 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 13:37:51.287479 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.287462 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rxggx\"" Jul 08 13:37:51.291747 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.291731 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.296099 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.296080 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-8mtsb\"" Jul 08 13:37:51.300678 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.300663 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-xfwk6\"" Jul 08 13:37:51.307511 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.307496 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 13:37:51.312025 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.312011 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 08 13:37:51.312114 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.312101 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.313075 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.313062 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.314640 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.314628 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.316480 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.316463 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.334419 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.334399 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.341219 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.341204 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.343271 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.343253 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.349050 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.349036 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.359693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.359676 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 08 13:37:51.379801 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.379781 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kj4bg\"" Jul 08 13:37:51.399900 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.399879 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.426816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.426794 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 13:37:51.440227 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.440204 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5pd96\"" Jul 08 13:37:51.460413 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.460346 2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:37:51.480515 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.480493 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.500887 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.500865 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-lc5hv\"" Jul 08 13:37:51.520644 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.520621 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dfn6w\"" Jul 08 13:37:51.541018 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.540998 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 13:37:51.559806 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.559786 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 13:37:51.579805 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.579785 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-pfpsg\"" Jul 08 13:37:51.600100 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.600078 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.620452 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.620430 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 13:37:51.640386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.640361 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 13:37:51.659874 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.659846 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 13:37:51.679862 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.679839 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-z4knk\"" Jul 08 13:37:51.699956 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.699929 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.720276 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.720198 2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 13:37:51.740265 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.740245 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.760218 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.760192 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 08 13:37:51.779870 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.779846 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 08 13:37:51.799697 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.799675 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7qjwb\"" Jul 08 13:37:51.819567 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.819548 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 13:37:51.840772 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.840751 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.859989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.859964 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 13:37:51.879812 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.879791 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 13:37:51.900233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.900210 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 13:37:51.920311 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:37:51.920288 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 13:42:42.039757 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:42:42.039724 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:42:42.043017 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:42:42.042998 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:45:00.137816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.137780 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:45:00.141214 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.141167 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:45:00.154848 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.154820 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:45:00.182429 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.182398 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6pc\" (UniqueName: \"kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc\") pod \"maas-api-key-cleanup-29725305-wbkxn\" (UID: \"bdd1993f-176e-4456-83a7-6b474d848a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:45:00.283623 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.283585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6pc\" (UniqueName: \"kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc\") pod \"maas-api-key-cleanup-29725305-wbkxn\" (UID: \"bdd1993f-176e-4456-83a7-6b474d848a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:45:00.291613 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.291574 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6pc\" (UniqueName: \"kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc\") pod \"maas-api-key-cleanup-29725305-wbkxn\" (UID: \"bdd1993f-176e-4456-83a7-6b474d848a00\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:45:00.451509 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.451414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:45:00.797292 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.797259 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:45:00.798640 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:45:00.798613 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd1993f_176e_4456_83a7_6b474d848a00.slice/crio-9024f5a0df79892097cead56de3677303f666afe1f2dcf0336a9139c3dd2dec1 WatchSource:0}: Error finding container 9024f5a0df79892097cead56de3677303f666afe1f2dcf0336a9139c3dd2dec1: Status 404 returned error can't find the container with id 9024f5a0df79892097cead56de3677303f666afe1f2dcf0336a9139c3dd2dec1 Jul 08 13:45:00.800844 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.800829 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:45:00.881343 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:00.881311 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" event={"ID":"bdd1993f-176e-4456-83a7-6b474d848a00","Type":"ContainerStarted","Data":"9024f5a0df79892097cead56de3677303f666afe1f2dcf0336a9139c3dd2dec1"} Jul 08 13:45:03.900790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:03.900744 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" event={"ID":"bdd1993f-176e-4456-83a7-6b474d848a00","Type":"ContainerStarted","Data":"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd"} Jul 08 13:45:03.918036 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:45:03.917989 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" podStartSLOduration=1.6529833470000002 podStartE2EDuration="3.917973941s" totalImagesPullingTime="2.264990594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:45:03.915466783 +0000 UTC m=+1342.409186573" watchObservedRunningTime="2026-07-08 13:45:03.917973941 +0000 UTC m=+1342.411693730" Jul 08 13:46:00.011151 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:00.011113 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:46:00.011730 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:00.011371 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" podUID="bdd1993f-176e-4456-83a7-6b474d848a00" containerName="cleanup" containerID="cri-o://b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd" gracePeriod=30 Jul 08 13:46:30.212545 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.212516 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:46:30.316037 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.315950 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5l6pc\" (UniqueName: \"kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc\") pod \"bdd1993f-176e-4456-83a7-6b474d848a00\" (UID: \"bdd1993f-176e-4456-83a7-6b474d848a00\") " Jul 08 13:46:30.318137 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.318103 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc" (OuterVolumeSpecName: "kube-api-access-5l6pc") pod "bdd1993f-176e-4456-83a7-6b474d848a00" (UID: "bdd1993f-176e-4456-83a7-6b474d848a00"). InnerVolumeSpecName "kube-api-access-5l6pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:46:30.340852 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.340823 2576 generic.go:356] "Generic (PLEG): container finished" podID="bdd1993f-176e-4456-83a7-6b474d848a00" containerID="b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd" exitCode=137 Jul 08 13:46:30.340972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.340882 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" Jul 08 13:46:30.340972 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.340904 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" event={"ID":"bdd1993f-176e-4456-83a7-6b474d848a00","Type":"ContainerDied","Data":"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd"} Jul 08 13:46:30.341048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.340981 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn" event={"ID":"bdd1993f-176e-4456-83a7-6b474d848a00","Type":"ContainerDied","Data":"9024f5a0df79892097cead56de3677303f666afe1f2dcf0336a9139c3dd2dec1"} Jul 08 13:46:30.341048 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.341003 2576 scope.go:117] "RemoveContainer" containerID="b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd" Jul 08 13:46:30.350314 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.350296 2576 scope.go:117] "RemoveContainer" containerID="b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd" Jul 08 13:46:30.350567 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:46:30.350547 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd\": container with ID starting with b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd not found: ID does not exist" containerID="b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd" Jul 08 13:46:30.350628 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.350575 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd"} err="failed to get container status \"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd\": rpc error: code = NotFound desc = could not find container \"b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd\": container with ID starting with b4cd4cdaf55ced8aaeedd694f54ec9c9d2092fdf112574390d32c964b9bf76fd not found: ID does not exist" Jul 08 13:46:30.362832 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.362794 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:46:30.365771 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.365748 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29725305-wbkxn"] Jul 08 13:46:30.417187 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:30.417154 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l6pc\" (UniqueName: \"kubernetes.io/projected/bdd1993f-176e-4456-83a7-6b474d848a00-kube-api-access-5l6pc\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:46:32.043630 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:46:32.043597 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd1993f-176e-4456-83a7-6b474d848a00" path="/var/lib/kubelet/pods/bdd1993f-176e-4456-83a7-6b474d848a00/volumes" Jul 08 13:47:42.070597 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:42.070487 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:47:42.076606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:42.076576 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:47:43.837732 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.837698 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:43.838126 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.838113 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd1993f-176e-4456-83a7-6b474d848a00" containerName="cleanup" Jul 08 13:47:43.838165 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.838127 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd1993f-176e-4456-83a7-6b474d848a00" containerName="cleanup" Jul 08 13:47:43.838222 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.838210 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd1993f-176e-4456-83a7-6b474d848a00" containerName="cleanup" Jul 08 13:47:43.840841 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.840824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.843128 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.843104 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-72d0489f-serving-cert\"" Jul 08 13:47:43.851887 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.851858 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:43.885575 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.885536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.885780 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.885621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9rph\" (UniqueName: \"kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.885780 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.885759 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.986397 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.986360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.986592 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.986415 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.986592 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.986448 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9rph\" (UniqueName: \"kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.986592 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:47:43.986573 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-serving-cert: secret "maas-api-e2e-ait-72d0489f-serving-cert" not found Jul 08 13:47:43.986781 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:47:43.986636 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls podName:e36d963b-b183-458d-ba45-a2162310531a nodeName:}" failed. No retries permitted until 2026-07-08 13:47:44.486618402 +0000 UTC m=+1502.980338170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls") pod "maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" (UID: "e36d963b-b183-458d-ba45-a2162310531a") : secret "maas-api-e2e-ait-72d0489f-serving-cert" not found Jul 08 13:47:43.987086 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.987063 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:43.997813 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:43.997784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9rph\" (UniqueName: \"kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:44.491780 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:44.491744 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:44.494266 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:44.494236 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") pod \"maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:44.753372 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:44.753291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:44.899375 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:44.899342 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:44.900338 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:47:44.900310 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode36d963b_b183_458d_ba45_a2162310531a.slice/crio-049408596ccc5812db6dd7c44dfd8045e8bad587813b22d616d3f8d8c798732d WatchSource:0}: Error finding container 049408596ccc5812db6dd7c44dfd8045e8bad587813b22d616d3f8d8c798732d: Status 404 returned error can't find the container with id 049408596ccc5812db6dd7c44dfd8045e8bad587813b22d616d3f8d8c798732d Jul 08 13:47:45.739046 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:45.739020 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" event={"ID":"e36d963b-b183-458d-ba45-a2162310531a","Type":"ContainerStarted","Data":"049408596ccc5812db6dd7c44dfd8045e8bad587813b22d616d3f8d8c798732d"} Jul 08 13:47:46.745873 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:46.745840 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" event={"ID":"e36d963b-b183-458d-ba45-a2162310531a","Type":"ContainerStarted","Data":"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f"} Jul 08 13:47:46.746286 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:46.745946 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:46.764951 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:46.764902 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" podStartSLOduration=2.99571632 podStartE2EDuration="3.764887666s" totalImagesPullingTime="769.171346ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:47:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:47:46.761314736 +0000 UTC m=+1505.255034525" watchObservedRunningTime="2026-07-08 13:47:46.764887666 +0000 UTC m=+1505.258607494" Jul 08 13:47:49.176604 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.176558 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:49.177093 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.176800 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" podUID="e36d963b-b183-458d-ba45-a2162310531a" containerName="maas-api" containerID="cri-o://33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f" gracePeriod=30 Jul 08 13:47:49.433515 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.433456 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:49.435884 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.435861 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9rph\" (UniqueName: \"kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph\") pod \"e36d963b-b183-458d-ba45-a2162310531a\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " Jul 08 13:47:49.436015 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.435933 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") pod \"e36d963b-b183-458d-ba45-a2162310531a\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " Jul 08 13:47:49.436015 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.435974 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca\") pod \"e36d963b-b183-458d-ba45-a2162310531a\" (UID: \"e36d963b-b183-458d-ba45-a2162310531a\") " Jul 08 13:47:49.436675 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.436362 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e36d963b-b183-458d-ba45-a2162310531a" (UID: "e36d963b-b183-458d-ba45-a2162310531a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:47:49.438107 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.438070 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph" (OuterVolumeSpecName: "kube-api-access-x9rph") pod "e36d963b-b183-458d-ba45-a2162310531a" (UID: "e36d963b-b183-458d-ba45-a2162310531a"). InnerVolumeSpecName "kube-api-access-x9rph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:47:49.438233 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.438146 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e36d963b-b183-458d-ba45-a2162310531a" (UID: "e36d963b-b183-458d-ba45-a2162310531a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:47:49.537601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.537564 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9rph\" (UniqueName: \"kubernetes.io/projected/e36d963b-b183-458d-ba45-a2162310531a-kube-api-access-x9rph\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:47:49.537601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.537594 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e36d963b-b183-458d-ba45-a2162310531a-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:47:49.537601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.537604 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e36d963b-b183-458d-ba45-a2162310531a-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:47:49.761793 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.761708 2576 generic.go:356] "Generic (PLEG): container finished" podID="e36d963b-b183-458d-ba45-a2162310531a" containerID="33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f" exitCode=0 Jul 08 13:47:49.761793 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.761753 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" Jul 08 13:47:49.761978 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.761788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" event={"ID":"e36d963b-b183-458d-ba45-a2162310531a","Type":"ContainerDied","Data":"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f"} Jul 08 13:47:49.761978 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.761839 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg" event={"ID":"e36d963b-b183-458d-ba45-a2162310531a","Type":"ContainerDied","Data":"049408596ccc5812db6dd7c44dfd8045e8bad587813b22d616d3f8d8c798732d"} Jul 08 13:47:49.761978 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.761863 2576 scope.go:117] "RemoveContainer" containerID="33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f" Jul 08 13:47:49.771464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.771447 2576 scope.go:117] "RemoveContainer" containerID="33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f" Jul 08 13:47:49.771728 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:47:49.771709 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f\": container with ID starting with 33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f not found: ID does not exist" containerID="33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f" Jul 08 13:47:49.771793 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.771734 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f"} err="failed to get container status \"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f\": rpc error: code = NotFound desc = could not find container \"33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f\": container with ID starting with 33d230b583643912b9d8c14f9263c6aa45f500819665647affe17750a1b13f6f not found: ID does not exist" Jul 08 13:47:49.783593 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.783568 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:49.786214 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:49.786191 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-72d0489f-6d4db57994-tqrcg"] Jul 08 13:47:50.043668 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:50.043582 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e36d963b-b183-458d-ba45-a2162310531a" path="/var/lib/kubelet/pods/e36d963b-b183-458d-ba45-a2162310531a/volumes" Jul 08 13:47:56.218064 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.218030 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:47:56.218653 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.218633 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e36d963b-b183-458d-ba45-a2162310531a" containerName="maas-api" Jul 08 13:47:56.218653 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.218655 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e36d963b-b183-458d-ba45-a2162310531a" containerName="maas-api" Jul 08 13:47:56.218808 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.218771 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e36d963b-b183-458d-ba45-a2162310531a" containerName="maas-api" Jul 08 13:47:56.221962 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.221943 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.224230 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.224207 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bf8c325b-serving-cert\"" Jul 08 13:47:56.233306 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.233280 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:47:56.298464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.298434 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.298670 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.298501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.298670 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.298581 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cjj\" (UniqueName: \"kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.407161 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.406333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cjj\" (UniqueName: \"kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.407161 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.406465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.407161 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.406535 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.407161 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:47:56.407147 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-serving-cert: secret "maas-api-e2e-ait-bf8c325b-serving-cert" not found Jul 08 13:47:56.407557 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:47:56.407249 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls podName:17330128-017c-4bd3-9fe2-eff95a98d078 nodeName:}" failed. No retries permitted until 2026-07-08 13:47:56.90722139 +0000 UTC m=+1515.400941158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls") pod "maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" (UID: "17330128-017c-4bd3-9fe2-eff95a98d078") : secret "maas-api-e2e-ait-bf8c325b-serving-cert" not found Jul 08 13:47:56.407679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.407504 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.415901 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.415878 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cjj\" (UniqueName: \"kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.911594 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.911555 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:56.914020 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:56.913986 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") pod \"maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:57.134693 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.134649 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:57.281759 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.281730 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:47:57.282995 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:47:57.282971 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17330128_017c_4bd3_9fe2_eff95a98d078.slice/crio-d6b3c12d81473fa5024fa3a6604dba641441a04b52f547a854dce76020d9a28f WatchSource:0}: Error finding container d6b3c12d81473fa5024fa3a6604dba641441a04b52f547a854dce76020d9a28f: Status 404 returned error can't find the container with id d6b3c12d81473fa5024fa3a6604dba641441a04b52f547a854dce76020d9a28f Jul 08 13:47:57.802374 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.802334 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" event={"ID":"17330128-017c-4bd3-9fe2-eff95a98d078","Type":"ContainerStarted","Data":"ac028a42a08ced3a0db48470e2794bab85a3bba6b057280d766ee718618d05f7"} Jul 08 13:47:57.802836 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.802393 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" event={"ID":"17330128-017c-4bd3-9fe2-eff95a98d078","Type":"ContainerStarted","Data":"d6b3c12d81473fa5024fa3a6604dba641441a04b52f547a854dce76020d9a28f"} Jul 08 13:47:57.802836 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.802423 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:47:57.834100 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:47:57.834049 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" podStartSLOduration=1.472482556 podStartE2EDuration="1.834034305s" totalImagesPullingTime="361.551749ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:47:57.83050323 +0000 UTC m=+1516.324223019" watchObservedRunningTime="2026-07-08 13:47:57.834034305 +0000 UTC m=+1516.327754094" Jul 08 13:48:01.687714 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:01.687678 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:48:01.688205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:01.687909 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" podUID="17330128-017c-4bd3-9fe2-eff95a98d078" containerName="maas-api" containerID="cri-o://ac028a42a08ced3a0db48470e2794bab85a3bba6b057280d766ee718618d05f7" gracePeriod=30 Jul 08 13:48:01.823601 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:01.823571 2576 generic.go:356] "Generic (PLEG): container finished" podID="17330128-017c-4bd3-9fe2-eff95a98d078" containerID="ac028a42a08ced3a0db48470e2794bab85a3bba6b057280d766ee718618d05f7" exitCode=0 Jul 08 13:48:01.823747 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:01.823608 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" event={"ID":"17330128-017c-4bd3-9fe2-eff95a98d078","Type":"ContainerDied","Data":"ac028a42a08ced3a0db48470e2794bab85a3bba6b057280d766ee718618d05f7"} Jul 08 13:48:01.951918 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:01.951853 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:48:02.057843 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.057814 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") pod \"17330128-017c-4bd3-9fe2-eff95a98d078\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " Jul 08 13:48:02.058010 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.057875 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca\") pod \"17330128-017c-4bd3-9fe2-eff95a98d078\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " Jul 08 13:48:02.058010 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.057973 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2cjj\" (UniqueName: \"kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj\") pod \"17330128-017c-4bd3-9fe2-eff95a98d078\" (UID: \"17330128-017c-4bd3-9fe2-eff95a98d078\") " Jul 08 13:48:02.058246 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.058223 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "17330128-017c-4bd3-9fe2-eff95a98d078" (UID: "17330128-017c-4bd3-9fe2-eff95a98d078"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:48:02.059908 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.059876 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "17330128-017c-4bd3-9fe2-eff95a98d078" (UID: "17330128-017c-4bd3-9fe2-eff95a98d078"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:48:02.060011 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.059911 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj" (OuterVolumeSpecName: "kube-api-access-w2cjj") pod "17330128-017c-4bd3-9fe2-eff95a98d078" (UID: "17330128-017c-4bd3-9fe2-eff95a98d078"). InnerVolumeSpecName "kube-api-access-w2cjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:48:02.159609 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.159567 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2cjj\" (UniqueName: \"kubernetes.io/projected/17330128-017c-4bd3-9fe2-eff95a98d078-kube-api-access-w2cjj\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:02.159609 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.159609 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17330128-017c-4bd3-9fe2-eff95a98d078-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:02.159819 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.159625 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17330128-017c-4bd3-9fe2-eff95a98d078-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:02.831339 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.831310 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" Jul 08 13:48:02.831339 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.831327 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5" event={"ID":"17330128-017c-4bd3-9fe2-eff95a98d078","Type":"ContainerDied","Data":"d6b3c12d81473fa5024fa3a6604dba641441a04b52f547a854dce76020d9a28f"} Jul 08 13:48:02.831796 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.831411 2576 scope.go:117] "RemoveContainer" containerID="ac028a42a08ced3a0db48470e2794bab85a3bba6b057280d766ee718618d05f7" Jul 08 13:48:02.853705 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.853668 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:48:02.857386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:02.857367 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bf8c325b-87c64d4f7-r8jk5"] Jul 08 13:48:04.044049 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:04.044013 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17330128-017c-4bd3-9fe2-eff95a98d078" path="/var/lib/kubelet/pods/17330128-017c-4bd3-9fe2-eff95a98d078/volumes" Jul 08 13:48:14.541786 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.541747 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:14.542376 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.542359 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17330128-017c-4bd3-9fe2-eff95a98d078" containerName="maas-api" Jul 08 13:48:14.542442 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.542379 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="17330128-017c-4bd3-9fe2-eff95a98d078" containerName="maas-api" Jul 08 13:48:14.542517 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.542506 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="17330128-017c-4bd3-9fe2-eff95a98d078" containerName="maas-api" Jul 08 13:48:14.545865 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.545844 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.548370 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.548351 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-50abd907-serving-cert\"" Jul 08 13:48:14.553716 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.553691 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:14.673493 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.673461 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.673686 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.673513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.673686 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.673543 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbb4t\" (UniqueName: \"kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.775013 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.774982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.775205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.775025 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.775205 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.775070 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbb4t\" (UniqueName: \"kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.775205 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:48:14.775129 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-serving-cert: secret "maas-api-e2e-derive-50abd907-serving-cert" not found Jul 08 13:48:14.775347 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:48:14.775210 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls podName:333f80a5-0b84-4516-b13e-6573e57584f1 nodeName:}" failed. No retries permitted until 2026-07-08 13:48:15.275190255 +0000 UTC m=+1533.768910027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls") pod "maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" (UID: "333f80a5-0b84-4516-b13e-6573e57584f1") : secret "maas-api-e2e-derive-50abd907-serving-cert" not found Jul 08 13:48:14.775742 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.775720 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:14.791636 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:14.791608 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbb4t\" (UniqueName: \"kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:15.279759 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:15.279718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:15.282251 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:15.282226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") pod \"maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:15.459212 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:15.459147 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:15.601833 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:15.601803 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:15.603075 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:48:15.603043 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333f80a5_0b84_4516_b13e_6573e57584f1.slice/crio-f6698aa64512ece9f38243ef7dbc6203988d3e34b212990de89b7784372e2076 WatchSource:0}: Error finding container f6698aa64512ece9f38243ef7dbc6203988d3e34b212990de89b7784372e2076: Status 404 returned error can't find the container with id f6698aa64512ece9f38243ef7dbc6203988d3e34b212990de89b7784372e2076 Jul 08 13:48:15.898490 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:15.898454 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" event={"ID":"333f80a5-0b84-4516-b13e-6573e57584f1","Type":"ContainerStarted","Data":"f6698aa64512ece9f38243ef7dbc6203988d3e34b212990de89b7784372e2076"} Jul 08 13:48:16.904921 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:16.904883 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" event={"ID":"333f80a5-0b84-4516-b13e-6573e57584f1","Type":"ContainerStarted","Data":"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e"} Jul 08 13:48:16.905315 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:16.905001 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:16.923600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:16.923556 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" podStartSLOduration=2.407174746 podStartE2EDuration="2.923541087s" totalImagesPullingTime="516.366341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:48:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:48:16.922681972 +0000 UTC m=+1535.416401788" watchObservedRunningTime="2026-07-08 13:48:16.923541087 +0000 UTC m=+1535.417260875" Jul 08 13:48:19.196386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.196341 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:19.196829 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.196601 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" podUID="333f80a5-0b84-4516-b13e-6573e57584f1" containerName="maas-api" containerID="cri-o://c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e" gracePeriod=30 Jul 08 13:48:19.462368 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.462307 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:19.622018 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.621981 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca\") pod \"333f80a5-0b84-4516-b13e-6573e57584f1\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " Jul 08 13:48:19.622018 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.622022 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbb4t\" (UniqueName: \"kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t\") pod \"333f80a5-0b84-4516-b13e-6573e57584f1\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " Jul 08 13:48:19.622278 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.622050 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") pod \"333f80a5-0b84-4516-b13e-6573e57584f1\" (UID: \"333f80a5-0b84-4516-b13e-6573e57584f1\") " Jul 08 13:48:19.622549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.622523 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "333f80a5-0b84-4516-b13e-6573e57584f1" (UID: "333f80a5-0b84-4516-b13e-6573e57584f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:48:19.624247 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.624225 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t" (OuterVolumeSpecName: "kube-api-access-jbb4t") pod "333f80a5-0b84-4516-b13e-6573e57584f1" (UID: "333f80a5-0b84-4516-b13e-6573e57584f1"). InnerVolumeSpecName "kube-api-access-jbb4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:48:19.624343 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.624259 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "333f80a5-0b84-4516-b13e-6573e57584f1" (UID: "333f80a5-0b84-4516-b13e-6573e57584f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:48:19.723475 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.723390 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/333f80a5-0b84-4516-b13e-6573e57584f1-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:19.723475 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.723418 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbb4t\" (UniqueName: \"kubernetes.io/projected/333f80a5-0b84-4516-b13e-6573e57584f1-kube-api-access-jbb4t\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:19.723475 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.723429 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/333f80a5-0b84-4516-b13e-6573e57584f1-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:48:19.922431 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.922398 2576 generic.go:356] "Generic (PLEG): container finished" podID="333f80a5-0b84-4516-b13e-6573e57584f1" containerID="c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e" exitCode=0 Jul 08 13:48:19.922600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.922463 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" Jul 08 13:48:19.922600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.922481 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" event={"ID":"333f80a5-0b84-4516-b13e-6573e57584f1","Type":"ContainerDied","Data":"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e"} Jul 08 13:48:19.922600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.922535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn" event={"ID":"333f80a5-0b84-4516-b13e-6573e57584f1","Type":"ContainerDied","Data":"f6698aa64512ece9f38243ef7dbc6203988d3e34b212990de89b7784372e2076"} Jul 08 13:48:19.922600 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.922557 2576 scope.go:117] "RemoveContainer" containerID="c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e" Jul 08 13:48:19.932012 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.931995 2576 scope.go:117] "RemoveContainer" containerID="c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e" Jul 08 13:48:19.932310 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:48:19.932287 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e\": container with ID starting with c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e not found: ID does not exist" containerID="c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e" Jul 08 13:48:19.932414 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.932320 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e"} err="failed to get container status \"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e\": rpc error: code = NotFound desc = could not find container \"c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e\": container with ID starting with c3dd92ab8225669ac6f8fa1a9b7455df572f130670d1df2901f3d00267862f3e not found: ID does not exist" Jul 08 13:48:19.944999 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.944945 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:19.947784 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:19.947761 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-50abd907-5bd4fc7479-v5lbn"] Jul 08 13:48:20.050938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:48:20.050859 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="333f80a5-0b84-4516-b13e-6573e57584f1" path="/var/lib/kubelet/pods/333f80a5-0b84-4516-b13e-6573e57584f1/volumes" Jul 08 13:50:04.444574 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.444496 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:04.445359 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.445121 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="333f80a5-0b84-4516-b13e-6573e57584f1" containerName="maas-api" Jul 08 13:50:04.445359 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.445136 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="333f80a5-0b84-4516-b13e-6573e57584f1" containerName="maas-api" Jul 08 13:50:04.445359 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.445270 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="333f80a5-0b84-4516-b13e-6573e57584f1" containerName="maas-api" Jul 08 13:50:04.448852 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.448832 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.451679 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.451657 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e91b2474-serving-cert\"" Jul 08 13:50:04.460832 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.460795 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:04.479603 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.479577 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.479757 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.479619 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.479757 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.479648 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwcm\" (UniqueName: \"kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.580512 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.580477 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.580698 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.580525 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.580698 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.580546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwcm\" (UniqueName: \"kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.580698 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:50:04.580656 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-serving-cert: secret "maas-api-e2e-mt-e91b2474-serving-cert" not found Jul 08 13:50:04.580852 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:50:04.580724 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls podName:cf14128c-c233-4bec-8bb0-84463fb7a29a nodeName:}" failed. No retries permitted until 2026-07-08 13:50:05.080704289 +0000 UTC m=+1643.574424061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls") pod "maas-api-e2e-mt-e91b2474-64598f6597-wlt68" (UID: "cf14128c-c233-4bec-8bb0-84463fb7a29a") : secret "maas-api-e2e-mt-e91b2474-serving-cert" not found Jul 08 13:50:04.581324 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.581299 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:04.592945 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:04.592909 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwcm\" (UniqueName: \"kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:05.086023 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:05.085986 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:05.088604 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:05.088573 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") pod \"maas-api-e2e-mt-e91b2474-64598f6597-wlt68\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:05.361991 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:05.361913 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:05.509086 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:05.509058 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:05.510785 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:50:05.510759 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf14128c_c233_4bec_8bb0_84463fb7a29a.slice/crio-67d6809f1574153a4ea30444f2d81c55929948f9b71799a4c5153fce1811fe67 WatchSource:0}: Error finding container 67d6809f1574153a4ea30444f2d81c55929948f9b71799a4c5153fce1811fe67: Status 404 returned error can't find the container with id 67d6809f1574153a4ea30444f2d81c55929948f9b71799a4c5153fce1811fe67 Jul 08 13:50:05.512082 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:05.512065 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:50:06.460571 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:06.460535 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" event={"ID":"cf14128c-c233-4bec-8bb0-84463fb7a29a","Type":"ContainerStarted","Data":"67d6809f1574153a4ea30444f2d81c55929948f9b71799a4c5153fce1811fe67"} Jul 08 13:50:07.466783 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:07.466736 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" event={"ID":"cf14128c-c233-4bec-8bb0-84463fb7a29a","Type":"ContainerStarted","Data":"210fbcd371f538078fd2cc71554e328f7c3d5f1a9096b7c97431194f5b389b4f"} Jul 08 13:50:07.467248 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:07.466898 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:07.484115 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:07.484068 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" podStartSLOduration=2.3802448099999998 podStartE2EDuration="3.484054362s" totalImagesPullingTime="1.103809552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:50:07.482912316 +0000 UTC m=+1645.976632106" watchObservedRunningTime="2026-07-08 13:50:07.484054362 +0000 UTC m=+1645.977774150" Jul 08 13:50:13.478776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:13.478744 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:16.366694 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.366655 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:16.367110 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.366916 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" podUID="cf14128c-c233-4bec-8bb0-84463fb7a29a" containerName="maas-api" containerID="cri-o://210fbcd371f538078fd2cc71554e328f7c3d5f1a9096b7c97431194f5b389b4f" gracePeriod=30 Jul 08 13:50:16.515282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.515235 2576 generic.go:356] "Generic (PLEG): container finished" podID="cf14128c-c233-4bec-8bb0-84463fb7a29a" containerID="210fbcd371f538078fd2cc71554e328f7c3d5f1a9096b7c97431194f5b389b4f" exitCode=0 Jul 08 13:50:16.515453 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.515423 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" event={"ID":"cf14128c-c233-4bec-8bb0-84463fb7a29a","Type":"ContainerDied","Data":"210fbcd371f538078fd2cc71554e328f7c3d5f1a9096b7c97431194f5b389b4f"} Jul 08 13:50:16.649696 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.649672 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:16.692933 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.692897 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvwcm\" (UniqueName: \"kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm\") pod \"cf14128c-c233-4bec-8bb0-84463fb7a29a\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " Jul 08 13:50:16.693113 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.692977 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") pod \"cf14128c-c233-4bec-8bb0-84463fb7a29a\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " Jul 08 13:50:16.693113 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.693027 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca\") pod \"cf14128c-c233-4bec-8bb0-84463fb7a29a\" (UID: \"cf14128c-c233-4bec-8bb0-84463fb7a29a\") " Jul 08 13:50:16.693440 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.693411 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf14128c-c233-4bec-8bb0-84463fb7a29a" (UID: "cf14128c-c233-4bec-8bb0-84463fb7a29a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:50:16.695092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.695066 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf14128c-c233-4bec-8bb0-84463fb7a29a" (UID: "cf14128c-c233-4bec-8bb0-84463fb7a29a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:50:16.695235 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.695213 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm" (OuterVolumeSpecName: "kube-api-access-rvwcm") pod "cf14128c-c233-4bec-8bb0-84463fb7a29a" (UID: "cf14128c-c233-4bec-8bb0-84463fb7a29a"). InnerVolumeSpecName "kube-api-access-rvwcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:50:16.794663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.794628 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf14128c-c233-4bec-8bb0-84463fb7a29a-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:50:16.794663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.794658 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf14128c-c233-4bec-8bb0-84463fb7a29a-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:50:16.794663 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:16.794668 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvwcm\" (UniqueName: \"kubernetes.io/projected/cf14128c-c233-4bec-8bb0-84463fb7a29a-kube-api-access-rvwcm\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:50:17.521187 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:17.521135 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" event={"ID":"cf14128c-c233-4bec-8bb0-84463fb7a29a","Type":"ContainerDied","Data":"67d6809f1574153a4ea30444f2d81c55929948f9b71799a4c5153fce1811fe67"} Jul 08 13:50:17.521187 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:17.521158 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68" Jul 08 13:50:17.521671 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:17.521220 2576 scope.go:117] "RemoveContainer" containerID="210fbcd371f538078fd2cc71554e328f7c3d5f1a9096b7c97431194f5b389b4f" Jul 08 13:50:17.546092 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:50:17.546058 2576 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf14128c_c233_4bec_8bb0_84463fb7a29a.slice\": RecentStats: unable to find data in memory cache]" Jul 08 13:50:17.547977 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:17.547934 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:17.552546 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:17.552518 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e91b2474-64598f6597-wlt68"] Jul 08 13:50:18.043424 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:18.043388 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf14128c-c233-4bec-8bb0-84463fb7a29a" path="/var/lib/kubelet/pods/cf14128c-c233-4bec-8bb0-84463fb7a29a/volumes" Jul 08 13:50:45.852351 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.852319 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:50:45.852760 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.852741 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf14128c-c233-4bec-8bb0-84463fb7a29a" containerName="maas-api" Jul 08 13:50:45.852760 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.852752 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf14128c-c233-4bec-8bb0-84463fb7a29a" containerName="maas-api" Jul 08 13:50:45.852832 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.852826 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf14128c-c233-4bec-8bb0-84463fb7a29a" containerName="maas-api" Jul 08 13:50:45.856027 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.856010 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.858605 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.858585 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-54e722cc-serving-cert\"" Jul 08 13:50:45.875347 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.875322 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.875491 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.875358 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.875491 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.875457 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drvrd\" (UniqueName: \"kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.880550 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.880523 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:50:45.976581 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.976548 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drvrd\" (UniqueName: \"kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.976798 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.976640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.976798 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.976671 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.976932 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:50:45.976805 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-serving-cert: secret "maas-api-e2e-mt-54e722cc-serving-cert" not found Jul 08 13:50:45.976932 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:50:45.976885 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls podName:2defc270-d361-428c-a580-55d5b3ac9621 nodeName:}" failed. No retries permitted until 2026-07-08 13:50:46.476864212 +0000 UTC m=+1684.970583982 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls") pod "maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" (UID: "2defc270-d361-428c-a580-55d5b3ac9621") : secret "maas-api-e2e-mt-54e722cc-serving-cert" not found Jul 08 13:50:45.977337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.977319 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:45.988914 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:45.988882 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drvrd\" (UniqueName: \"kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:46.481635 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:46.481595 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:46.484078 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:46.484044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") pod \"maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:46.766371 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:46.766284 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:46.914344 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:46.914315 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:50:46.916551 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:50:46.916521 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2defc270_d361_428c_a580_55d5b3ac9621.slice/crio-1cdc50ba812cf8630cfbf5f6dd58cb17cda003c4761c6ed445fd44dd243b0c5a WatchSource:0}: Error finding container 1cdc50ba812cf8630cfbf5f6dd58cb17cda003c4761c6ed445fd44dd243b0c5a: Status 404 returned error can't find the container with id 1cdc50ba812cf8630cfbf5f6dd58cb17cda003c4761c6ed445fd44dd243b0c5a Jul 08 13:50:47.680407 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:47.680370 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" event={"ID":"2defc270-d361-428c-a580-55d5b3ac9621","Type":"ContainerStarted","Data":"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461"} Jul 08 13:50:47.680577 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:47.680447 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" event={"ID":"2defc270-d361-428c-a580-55d5b3ac9621","Type":"ContainerStarted","Data":"1cdc50ba812cf8630cfbf5f6dd58cb17cda003c4761c6ed445fd44dd243b0c5a"} Jul 08 13:50:47.680577 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:47.680513 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:50:47.701976 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:47.701924 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" podStartSLOduration=2.320757123 podStartE2EDuration="2.701907345s" totalImagesPullingTime="381.150222ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:50:47.700723982 +0000 UTC m=+1686.194443771" watchObservedRunningTime="2026-07-08 13:50:47.701907345 +0000 UTC m=+1686.195627134" Jul 08 13:50:53.690790 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:50:53.690754 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:51:12.842136 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:12.842101 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:51:12.842647 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:12.842384 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" podUID="2defc270-d361-428c-a580-55d5b3ac9621" containerName="maas-api" containerID="cri-o://ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461" gracePeriod=30 Jul 08 13:51:13.099917 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.099854 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:51:13.240857 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.240819 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drvrd\" (UniqueName: \"kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd\") pod \"2defc270-d361-428c-a580-55d5b3ac9621\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " Jul 08 13:51:13.240857 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.240867 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca\") pod \"2defc270-d361-428c-a580-55d5b3ac9621\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " Jul 08 13:51:13.241058 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.240899 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") pod \"2defc270-d361-428c-a580-55d5b3ac9621\" (UID: \"2defc270-d361-428c-a580-55d5b3ac9621\") " Jul 08 13:51:13.241284 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.241255 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2defc270-d361-428c-a580-55d5b3ac9621" (UID: "2defc270-d361-428c-a580-55d5b3ac9621"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:51:13.243092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.243055 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2defc270-d361-428c-a580-55d5b3ac9621" (UID: "2defc270-d361-428c-a580-55d5b3ac9621"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:51:13.243092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.243063 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd" (OuterVolumeSpecName: "kube-api-access-drvrd") pod "2defc270-d361-428c-a580-55d5b3ac9621" (UID: "2defc270-d361-428c-a580-55d5b3ac9621"). InnerVolumeSpecName "kube-api-access-drvrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:51:13.341607 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.341566 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drvrd\" (UniqueName: \"kubernetes.io/projected/2defc270-d361-428c-a580-55d5b3ac9621-kube-api-access-drvrd\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:51:13.341607 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.341597 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2defc270-d361-428c-a580-55d5b3ac9621-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:51:13.341607 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.341611 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2defc270-d361-428c-a580-55d5b3ac9621-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:51:13.816040 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.816002 2576 generic.go:356] "Generic (PLEG): container finished" podID="2defc270-d361-428c-a580-55d5b3ac9621" containerID="ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461" exitCode=0 Jul 08 13:51:13.816337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.816062 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" Jul 08 13:51:13.816337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.816087 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" event={"ID":"2defc270-d361-428c-a580-55d5b3ac9621","Type":"ContainerDied","Data":"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461"} Jul 08 13:51:13.816337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.816165 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn" event={"ID":"2defc270-d361-428c-a580-55d5b3ac9621","Type":"ContainerDied","Data":"1cdc50ba812cf8630cfbf5f6dd58cb17cda003c4761c6ed445fd44dd243b0c5a"} Jul 08 13:51:13.816337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.816206 2576 scope.go:117] "RemoveContainer" containerID="ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461" Jul 08 13:51:13.827613 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.827573 2576 scope.go:117] "RemoveContainer" containerID="ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461" Jul 08 13:51:13.828073 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:51:13.828052 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461\": container with ID starting with ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461 not found: ID does not exist" containerID="ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461" Jul 08 13:51:13.828158 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.828085 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461"} err="failed to get container status \"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461\": rpc error: code = NotFound desc = could not find container \"ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461\": container with ID starting with ea26216562fa34d6b02f88a847fa46d0a1719160fbaff276102598c21de2e461 not found: ID does not exist" Jul 08 13:51:13.840398 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.840374 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:51:13.842903 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:13.842876 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-54e722cc-6b577f8cc6-xqkgn"] Jul 08 13:51:14.043342 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:51:14.043311 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2defc270-d361-428c-a580-55d5b3ac9621" path="/var/lib/kubelet/pods/2defc270-d361-428c-a580-55d5b3ac9621/volumes" Jul 08 13:52:42.100584 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:52:42.100475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:52:42.109222 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:52:42.109199 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:53:10.396352 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.396319 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:53:10.396840 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.396764 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2defc270-d361-428c-a580-55d5b3ac9621" containerName="maas-api" Jul 08 13:53:10.396840 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.396777 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2defc270-d361-428c-a580-55d5b3ac9621" containerName="maas-api" Jul 08 13:53:10.396957 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.396856 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2defc270-d361-428c-a580-55d5b3ac9621" containerName="maas-api" Jul 08 13:53:10.400435 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.400411 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.402977 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.402953 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ac32a0-serving-cert\"" Jul 08 13:53:10.410052 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.410025 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:53:10.520229 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.520187 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.520403 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.520249 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qlj7\" (UniqueName: \"kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.520403 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.520270 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.621591 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.621550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.621801 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.621600 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qlj7\" (UniqueName: \"kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.621801 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.621625 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.621801 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:53:10.621743 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-serving-cert: secret "maas-api-e2e-inf-a-ac32a0-serving-cert" not found Jul 08 13:53:10.622057 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:53:10.621812 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls podName:933df821-a809-492e-b48b-eb37c8ed3c94 nodeName:}" failed. No retries permitted until 2026-07-08 13:53:11.121791231 +0000 UTC m=+1829.615510999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls") pod "maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" (UID: "933df821-a809-492e-b48b-eb37c8ed3c94") : secret "maas-api-e2e-inf-a-ac32a0-serving-cert" not found Jul 08 13:53:10.622376 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.622352 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:10.635332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:10.635305 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qlj7\" (UniqueName: \"kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:11.126643 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:11.126604 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:11.129194 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:11.129156 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") pod \"maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:11.312446 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:11.312412 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:11.457887 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:11.457857 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:53:11.458600 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:53:11.458572 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933df821_a809_492e_b48b_eb37c8ed3c94.slice/crio-8a4de5f1507be4d4b8dd520681083b29dc915dd53c42f0c5428e49c1800d312d WatchSource:0}: Error finding container 8a4de5f1507be4d4b8dd520681083b29dc915dd53c42f0c5428e49c1800d312d: Status 404 returned error can't find the container with id 8a4de5f1507be4d4b8dd520681083b29dc915dd53c42f0c5428e49c1800d312d Jul 08 13:53:12.411827 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:12.411793 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" event={"ID":"933df821-a809-492e-b48b-eb37c8ed3c94","Type":"ContainerStarted","Data":"8a4de5f1507be4d4b8dd520681083b29dc915dd53c42f0c5428e49c1800d312d"} Jul 08 13:53:14.423612 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:14.423576 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" event={"ID":"933df821-a809-492e-b48b-eb37c8ed3c94","Type":"ContainerStarted","Data":"20f655b1c11b540a43243025e9c754c7e582b2e2c8bda6f249081b7ba05689d9"} Jul 08 13:53:14.424072 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:14.423687 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:14.443263 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:14.443216 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" podStartSLOduration=2.424222714 podStartE2EDuration="4.443200424s" totalImagesPullingTime="2.01897771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:53:14.441016324 +0000 UTC m=+1832.934736275" watchObservedRunningTime="2026-07-08 13:53:14.443200424 +0000 UTC m=+1832.936920213" Jul 08 13:53:18.534340 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.534289 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:53:18.538521 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.538497 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.540777 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.540752 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8f6d76-serving-cert\"" Jul 08 13:53:18.549407 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.549376 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:53:18.696068 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.696033 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.696269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.696119 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.696269 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.696139 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdnjz\" (UniqueName: \"kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.797632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.797546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.797632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.797583 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdnjz\" (UniqueName: \"kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.797632 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.797631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.798257 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.798238 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.800014 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.799994 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.806753 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.806730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdnjz\" (UniqueName: \"kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz\") pod \"maas-api-e2e-inf-b-8f6d76-555c579849-mgl72\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:18.853564 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:18.853532 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:19.007369 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:19.007341 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:53:19.009855 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:53:19.009828 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbfd1da_afb0_44b0_b8bd_c49517462e5a.slice/crio-5e366b0b4568f4bd60a0b553a203620a7ffb39e342f72fe537610322146b3bf4 WatchSource:0}: Error finding container 5e366b0b4568f4bd60a0b553a203620a7ffb39e342f72fe537610322146b3bf4: Status 404 returned error can't find the container with id 5e366b0b4568f4bd60a0b553a203620a7ffb39e342f72fe537610322146b3bf4 Jul 08 13:53:19.450989 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:19.450959 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" event={"ID":"bdbfd1da-afb0-44b0-b8bd-c49517462e5a","Type":"ContainerStarted","Data":"5e366b0b4568f4bd60a0b553a203620a7ffb39e342f72fe537610322146b3bf4"} Jul 08 13:53:20.435471 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:20.435441 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:53:20.458969 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:20.458905 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" event={"ID":"bdbfd1da-afb0-44b0-b8bd-c49517462e5a","Type":"ContainerStarted","Data":"a1a98565390ae63c010541a386e56f6ce143c3dce85cd3777cea19c5b18e0c81"} Jul 08 13:53:20.459193 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:20.459009 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:53:20.476262 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:20.476210 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" podStartSLOduration=2.04258673 podStartE2EDuration="2.476188407s" totalImagesPullingTime="433.601677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:53:20.474025248 +0000 UTC m=+1838.967745038" watchObservedRunningTime="2026-07-08 13:53:20.476188407 +0000 UTC m=+1838.969908201" Jul 08 13:53:26.470819 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:53:26.470788 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:54:43.761161 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:43.761126 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:54:43.761764 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:43.761521 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" podUID="933df821-a809-492e-b48b-eb37c8ed3c94" containerName="maas-api" containerID="cri-o://20f655b1c11b540a43243025e9c754c7e582b2e2c8bda6f249081b7ba05689d9" gracePeriod=30 Jul 08 13:54:43.902408 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:43.902377 2576 generic.go:356] "Generic (PLEG): container finished" podID="933df821-a809-492e-b48b-eb37c8ed3c94" containerID="20f655b1c11b540a43243025e9c754c7e582b2e2c8bda6f249081b7ba05689d9" exitCode=0 Jul 08 13:54:43.902596 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:43.902458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" event={"ID":"933df821-a809-492e-b48b-eb37c8ed3c94","Type":"ContainerDied","Data":"20f655b1c11b540a43243025e9c754c7e582b2e2c8bda6f249081b7ba05689d9"} Jul 08 13:54:44.014809 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.014747 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:54:44.092348 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.092310 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca\") pod \"933df821-a809-492e-b48b-eb37c8ed3c94\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " Jul 08 13:54:44.092541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.092409 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") pod \"933df821-a809-492e-b48b-eb37c8ed3c94\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " Jul 08 13:54:44.092541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.092441 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qlj7\" (UniqueName: \"kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7\") pod \"933df821-a809-492e-b48b-eb37c8ed3c94\" (UID: \"933df821-a809-492e-b48b-eb37c8ed3c94\") " Jul 08 13:54:44.092753 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.092724 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "933df821-a809-492e-b48b-eb37c8ed3c94" (UID: "933df821-a809-492e-b48b-eb37c8ed3c94"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:54:44.094568 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.094531 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "933df821-a809-492e-b48b-eb37c8ed3c94" (UID: "933df821-a809-492e-b48b-eb37c8ed3c94"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:54:44.094678 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.094568 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7" (OuterVolumeSpecName: "kube-api-access-6qlj7") pod "933df821-a809-492e-b48b-eb37c8ed3c94" (UID: "933df821-a809-492e-b48b-eb37c8ed3c94"). InnerVolumeSpecName "kube-api-access-6qlj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:54:44.194092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.194056 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/933df821-a809-492e-b48b-eb37c8ed3c94-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:54:44.194092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.194086 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/933df821-a809-492e-b48b-eb37c8ed3c94-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:54:44.194092 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.194096 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qlj7\" (UniqueName: \"kubernetes.io/projected/933df821-a809-492e-b48b-eb37c8ed3c94-kube-api-access-6qlj7\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:54:44.909467 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.909436 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" Jul 08 13:54:44.909954 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.909487 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k" event={"ID":"933df821-a809-492e-b48b-eb37c8ed3c94","Type":"ContainerDied","Data":"8a4de5f1507be4d4b8dd520681083b29dc915dd53c42f0c5428e49c1800d312d"} Jul 08 13:54:44.909954 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.909553 2576 scope.go:117] "RemoveContainer" containerID="20f655b1c11b540a43243025e9c754c7e582b2e2c8bda6f249081b7ba05689d9" Jul 08 13:54:44.931864 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.931828 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:54:44.934350 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:44.934329 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ac32a0-667c5d69ff-nvf7k"] Jul 08 13:54:46.044200 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:54:46.044131 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="933df821-a809-492e-b48b-eb37c8ed3c94" path="/var/lib/kubelet/pods/933df821-a809-492e-b48b-eb37c8ed3c94/volumes" Jul 08 13:55:26.756898 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:26.756862 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:55:26.757541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:26.757128 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" podUID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" containerName="maas-api" containerID="cri-o://a1a98565390ae63c010541a386e56f6ce143c3dce85cd3777cea19c5b18e0c81" gracePeriod=30 Jul 08 13:55:27.126376 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.126344 2576 generic.go:356] "Generic (PLEG): container finished" podID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" containerID="a1a98565390ae63c010541a386e56f6ce143c3dce85cd3777cea19c5b18e0c81" exitCode=0 Jul 08 13:55:27.126539 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.126390 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" event={"ID":"bdbfd1da-afb0-44b0-b8bd-c49517462e5a","Type":"ContainerDied","Data":"a1a98565390ae63c010541a386e56f6ce143c3dce85cd3777cea19c5b18e0c81"} Jul 08 13:55:27.216010 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.215986 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:55:27.277484 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.277453 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca\") pod \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " Jul 08 13:55:27.277662 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.277515 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls\") pod \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " Jul 08 13:55:27.277662 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.277590 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdnjz\" (UniqueName: \"kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz\") pod \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\" (UID: \"bdbfd1da-afb0-44b0-b8bd-c49517462e5a\") " Jul 08 13:55:27.277837 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.277802 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdbfd1da-afb0-44b0-b8bd-c49517462e5a" (UID: "bdbfd1da-afb0-44b0-b8bd-c49517462e5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:55:27.279572 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.279550 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdbfd1da-afb0-44b0-b8bd-c49517462e5a" (UID: "bdbfd1da-afb0-44b0-b8bd-c49517462e5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:55:27.279690 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.279665 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz" (OuterVolumeSpecName: "kube-api-access-sdnjz") pod "bdbfd1da-afb0-44b0-b8bd-c49517462e5a" (UID: "bdbfd1da-afb0-44b0-b8bd-c49517462e5a"). InnerVolumeSpecName "kube-api-access-sdnjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:55:27.378857 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.378772 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:27.378857 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.378804 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:27.378857 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:27.378817 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdnjz\" (UniqueName: \"kubernetes.io/projected/bdbfd1da-afb0-44b0-b8bd-c49517462e5a-kube-api-access-sdnjz\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:28.133202 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:28.133136 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" event={"ID":"bdbfd1da-afb0-44b0-b8bd-c49517462e5a","Type":"ContainerDied","Data":"5e366b0b4568f4bd60a0b553a203620a7ffb39e342f72fe537610322146b3bf4"} Jul 08 13:55:28.133645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:28.133219 2576 scope.go:117] "RemoveContainer" containerID="a1a98565390ae63c010541a386e56f6ce143c3dce85cd3777cea19c5b18e0c81" Jul 08 13:55:28.133645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:28.133157 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72" Jul 08 13:55:28.150357 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:28.150335 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:55:28.153159 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:28.153137 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8f6d76-555c579849-mgl72"] Jul 08 13:55:30.043657 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:30.043620 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" path="/var/lib/kubelet/pods/bdbfd1da-afb0-44b0-b8bd-c49517462e5a/volumes" Jul 08 13:55:37.230332 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230211 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230692 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="933df821-a809-492e-b48b-eb37c8ed3c94" containerName="maas-api" Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230706 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="933df821-a809-492e-b48b-eb37c8ed3c94" containerName="maas-api" Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230729 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" containerName="maas-api" Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230736 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" containerName="maas-api" Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230797 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="933df821-a809-492e-b48b-eb37c8ed3c94" containerName="maas-api" Jul 08 13:55:37.230820 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.230811 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbfd1da-afb0-44b0-b8bd-c49517462e5a" containerName="maas-api" Jul 08 13:55:37.235196 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.235161 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.238114 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.238095 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8dd341-serving-cert\"" Jul 08 13:55:37.250155 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.250129 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:55:37.375122 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.375080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.375313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.375138 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.375313 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.375201 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vg8f\" (UniqueName: \"kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.475950 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.475908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.475950 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.475955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.476251 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.475976 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vg8f\" (UniqueName: \"kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.476251 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:55:37.476088 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-serving-cert: secret "maas-api-e2e-api-a-8dd341-serving-cert" not found Jul 08 13:55:37.476251 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:55:37.476168 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls podName:98cad7be-e95d-4e1f-8e8c-1d4601954fc0 nodeName:}" failed. No retries permitted until 2026-07-08 13:55:37.976146722 +0000 UTC m=+1976.469866494 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls") pod "maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" (UID: "98cad7be-e95d-4e1f-8e8c-1d4601954fc0") : secret "maas-api-e2e-api-a-8dd341-serving-cert" not found Jul 08 13:55:37.476712 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.476688 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.489012 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.488953 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vg8f\" (UniqueName: \"kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.981302 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.981254 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:37.984077 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:37.984044 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") pod \"maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:38.146317 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:38.146285 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:38.289997 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:38.289971 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:55:38.292721 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:55:38.292690 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98cad7be_e95d_4e1f_8e8c_1d4601954fc0.slice/crio-a71439247af40b7b4cb9413dd718f13b36b13558e474cbbf70fcc0f668f78714 WatchSource:0}: Error finding container a71439247af40b7b4cb9413dd718f13b36b13558e474cbbf70fcc0f668f78714: Status 404 returned error can't find the container with id a71439247af40b7b4cb9413dd718f13b36b13558e474cbbf70fcc0f668f78714 Jul 08 13:55:38.294282 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:38.294261 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 13:55:39.193947 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:39.193906 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" event={"ID":"98cad7be-e95d-4e1f-8e8c-1d4601954fc0","Type":"ContainerStarted","Data":"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c"} Jul 08 13:55:39.194142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:39.193963 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" event={"ID":"98cad7be-e95d-4e1f-8e8c-1d4601954fc0","Type":"ContainerStarted","Data":"a71439247af40b7b4cb9413dd718f13b36b13558e474cbbf70fcc0f668f78714"} Jul 08 13:55:39.194142 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:39.194018 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:39.213761 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:39.213710 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" podStartSLOduration=1.7716150910000001 podStartE2EDuration="2.213694498s" totalImagesPullingTime="442.079407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:55:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:55:39.210384768 +0000 UTC m=+1977.704104581" watchObservedRunningTime="2026-07-08 13:55:39.213694498 +0000 UTC m=+1977.707414286" Jul 08 13:55:45.205439 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:45.205408 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:57.827295 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:57.827261 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:55:57.827804 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:57.827512 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" podUID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" containerName="maas-api" containerID="cri-o://c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c" gracePeriod=30 Jul 08 13:55:58.100913 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.100890 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:58.170491 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.170456 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") pod \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " Jul 08 13:55:58.170645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.170505 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vg8f\" (UniqueName: \"kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f\") pod \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " Jul 08 13:55:58.170645 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.170588 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca\") pod \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\" (UID: \"98cad7be-e95d-4e1f-8e8c-1d4601954fc0\") " Jul 08 13:55:58.170983 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.170957 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "98cad7be-e95d-4e1f-8e8c-1d4601954fc0" (UID: "98cad7be-e95d-4e1f-8e8c-1d4601954fc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:55:58.172704 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.172674 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "98cad7be-e95d-4e1f-8e8c-1d4601954fc0" (UID: "98cad7be-e95d-4e1f-8e8c-1d4601954fc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:55:58.172704 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.172682 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f" (OuterVolumeSpecName: "kube-api-access-5vg8f") pod "98cad7be-e95d-4e1f-8e8c-1d4601954fc0" (UID: "98cad7be-e95d-4e1f-8e8c-1d4601954fc0"). InnerVolumeSpecName "kube-api-access-5vg8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:55:58.272008 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.271974 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:58.272008 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.272002 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:58.272008 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.272013 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vg8f\" (UniqueName: \"kubernetes.io/projected/98cad7be-e95d-4e1f-8e8c-1d4601954fc0-kube-api-access-5vg8f\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:55:58.295517 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.295483 2576 generic.go:356] "Generic (PLEG): container finished" podID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" containerID="c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c" exitCode=0 Jul 08 13:55:58.295660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.295533 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" Jul 08 13:55:58.295660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.295562 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" event={"ID":"98cad7be-e95d-4e1f-8e8c-1d4601954fc0","Type":"ContainerDied","Data":"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c"} Jul 08 13:55:58.295660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.295627 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw" event={"ID":"98cad7be-e95d-4e1f-8e8c-1d4601954fc0","Type":"ContainerDied","Data":"a71439247af40b7b4cb9413dd718f13b36b13558e474cbbf70fcc0f668f78714"} Jul 08 13:55:58.295660 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.295659 2576 scope.go:117] "RemoveContainer" containerID="c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c" Jul 08 13:55:58.305888 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.305869 2576 scope.go:117] "RemoveContainer" containerID="c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c" Jul 08 13:55:58.306229 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:55:58.306193 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c\": container with ID starting with c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c not found: ID does not exist" containerID="c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c" Jul 08 13:55:58.306348 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.306239 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c"} err="failed to get container status \"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c\": rpc error: code = NotFound desc = could not find container \"c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c\": container with ID starting with c6872fde1cfb5c3c1450210c3f9e6b3f6a422f19e630953f163746008723ae2c not found: ID does not exist" Jul 08 13:55:58.345776 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.345679 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:55:58.359894 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:55:58.359866 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8dd341-85b964f7f4-zfkzw"] Jul 08 13:56:00.043133 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:00.043089 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" path="/var/lib/kubelet/pods/98cad7be-e95d-4e1f-8e8c-1d4601954fc0/volumes" Jul 08 13:56:15.404777 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.404741 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:56:15.405392 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.405373 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" containerName="maas-api" Jul 08 13:56:15.405464 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.405396 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" containerName="maas-api" Jul 08 13:56:15.405518 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.405507 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="98cad7be-e95d-4e1f-8e8c-1d4601954fc0" containerName="maas-api" Jul 08 13:56:15.410525 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.410505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.413200 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.413163 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-4f14d0-serving-cert\"" Jul 08 13:56:15.419855 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.419412 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:56:15.528124 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.528085 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqkn8\" (UniqueName: \"kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.528329 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.528245 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.528415 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.528334 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.629238 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.629167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.629480 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.629263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.629480 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.629308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqkn8\" (UniqueName: \"kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.629480 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:56:15.629425 2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-serving-cert: secret "maas-api-e2e-shared-a-4f14d0-serving-cert" not found Jul 08 13:56:15.629661 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:56:15.629515 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls podName:93aa133f-357b-4bdc-bca0-8967f9bcec65 nodeName:}" failed. No retries permitted until 2026-07-08 13:56:16.129492573 +0000 UTC m=+2014.623212357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls") pod "maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" (UID: "93aa133f-357b-4bdc-bca0-8967f9bcec65") : secret "maas-api-e2e-shared-a-4f14d0-serving-cert" not found Jul 08 13:56:15.629961 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.629937 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:15.639520 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:15.639498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqkn8\" (UniqueName: \"kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:16.134934 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:16.134897 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:16.137514 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:16.137487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") pod \"maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:16.324245 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:16.324205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:16.467132 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:16.467103 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:56:16.468756 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:56:16.468726 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93aa133f_357b_4bdc_bca0_8967f9bcec65.slice/crio-e5c80bdac362c6d0b039e07702d2d4391731537bf0cf7e0fa68044fcca0dda55 WatchSource:0}: Error finding container e5c80bdac362c6d0b039e07702d2d4391731537bf0cf7e0fa68044fcca0dda55: Status 404 returned error can't find the container with id e5c80bdac362c6d0b039e07702d2d4391731537bf0cf7e0fa68044fcca0dda55 Jul 08 13:56:17.396746 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:17.396712 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" event={"ID":"93aa133f-357b-4bdc-bca0-8967f9bcec65","Type":"ContainerStarted","Data":"e5c80bdac362c6d0b039e07702d2d4391731537bf0cf7e0fa68044fcca0dda55"} Jul 08 13:56:19.409104 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:19.409070 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" event={"ID":"93aa133f-357b-4bdc-bca0-8967f9bcec65","Type":"ContainerStarted","Data":"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123"} Jul 08 13:56:19.409606 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:19.409195 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:19.428333 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:19.428276 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" podStartSLOduration=2.545781011 podStartE2EDuration="4.428262038s" totalImagesPullingTime="1.882481027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:56:19.4254544 +0000 UTC m=+2017.919174186" watchObservedRunningTime="2026-07-08 13:56:19.428262038 +0000 UTC m=+2017.921981828" Jul 08 13:56:23.538997 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.538956 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:56:23.543571 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.543548 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.547069 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.547044 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-e225ef-serving-cert\"" Jul 08 13:56:23.550571 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.550546 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:56:23.600540 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.600468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52b6s\" (UniqueName: \"kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.600787 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.600568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.600787 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.600621 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.701528 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.701490 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.701528 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.701533 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.701738 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.701624 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52b6s\" (UniqueName: \"kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.702283 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.702255 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.703942 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.703915 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.711639 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.711618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52b6s\" (UniqueName: \"kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s\") pod \"maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:23.857709 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:23.857621 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:24.002373 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:24.002343 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:56:24.004231 ip-10-0-138-212 kubenswrapper[2576]: W0708 13:56:24.004195 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bcff7d0_dee6_4312_8835_f8fab8989e37.slice/crio-6682f3a82aeab02410d051c44fe467f5dcc254e05b57d69a2b5a33a3a01e683a WatchSource:0}: Error finding container 6682f3a82aeab02410d051c44fe467f5dcc254e05b57d69a2b5a33a3a01e683a: Status 404 returned error can't find the container with id 6682f3a82aeab02410d051c44fe467f5dcc254e05b57d69a2b5a33a3a01e683a Jul 08 13:56:24.436085 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:24.436058 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" event={"ID":"5bcff7d0-dee6-4312-8835-f8fab8989e37","Type":"ContainerStarted","Data":"6682f3a82aeab02410d051c44fe467f5dcc254e05b57d69a2b5a33a3a01e683a"} Jul 08 13:56:25.422132 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:25.422094 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:56:25.443270 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:25.443211 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" event={"ID":"5bcff7d0-dee6-4312-8835-f8fab8989e37","Type":"ContainerStarted","Data":"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8"} Jul 08 13:56:25.443565 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:25.443533 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:56:25.463549 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:25.463491 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" podStartSLOduration=2.077322313 podStartE2EDuration="2.463470684s" totalImagesPullingTime="386.148371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 13:56:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 13:56:25.461628671 +0000 UTC m=+2023.955348488" watchObservedRunningTime="2026-07-08 13:56:25.463470684 +0000 UTC m=+2023.957190475" Jul 08 13:56:31.454992 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:56:31.454957 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:57:42.131118 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:57:42.131000 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:57:42.141245 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:57:42.141221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 13:59:33.516626 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.516590 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:59:33.517149 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.516862 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" podUID="93aa133f-357b-4bdc-bca0-8967f9bcec65" containerName="maas-api" containerID="cri-o://7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123" gracePeriod=30 Jul 08 13:59:33.777264 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.777201 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:59:33.949541 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.949507 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") pod \"93aa133f-357b-4bdc-bca0-8967f9bcec65\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " Jul 08 13:59:33.949719 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.949591 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca\") pod \"93aa133f-357b-4bdc-bca0-8967f9bcec65\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " Jul 08 13:59:33.949719 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.949643 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqkn8\" (UniqueName: \"kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8\") pod \"93aa133f-357b-4bdc-bca0-8967f9bcec65\" (UID: \"93aa133f-357b-4bdc-bca0-8967f9bcec65\") " Jul 08 13:59:33.949938 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.949908 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "93aa133f-357b-4bdc-bca0-8967f9bcec65" (UID: "93aa133f-357b-4bdc-bca0-8967f9bcec65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:59:33.951699 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.951662 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "93aa133f-357b-4bdc-bca0-8967f9bcec65" (UID: "93aa133f-357b-4bdc-bca0-8967f9bcec65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:59:33.951808 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:33.951721 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8" (OuterVolumeSpecName: "kube-api-access-vqkn8") pod "93aa133f-357b-4bdc-bca0-8967f9bcec65" (UID: "93aa133f-357b-4bdc-bca0-8967f9bcec65"). InnerVolumeSpecName "kube-api-access-vqkn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:59:34.051105 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.051023 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/93aa133f-357b-4bdc-bca0-8967f9bcec65-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:34.051105 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.051054 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqkn8\" (UniqueName: \"kubernetes.io/projected/93aa133f-357b-4bdc-bca0-8967f9bcec65-kube-api-access-vqkn8\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:34.051105 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.051072 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/93aa133f-357b-4bdc-bca0-8967f9bcec65-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:34.429797 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.429757 2576 generic.go:356] "Generic (PLEG): container finished" podID="93aa133f-357b-4bdc-bca0-8967f9bcec65" containerID="7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123" exitCode=0 Jul 08 13:59:34.429797 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.429804 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" Jul 08 13:59:34.430062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.429855 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" event={"ID":"93aa133f-357b-4bdc-bca0-8967f9bcec65","Type":"ContainerDied","Data":"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123"} Jul 08 13:59:34.430062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.429915 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6" event={"ID":"93aa133f-357b-4bdc-bca0-8967f9bcec65","Type":"ContainerDied","Data":"e5c80bdac362c6d0b039e07702d2d4391731537bf0cf7e0fa68044fcca0dda55"} Jul 08 13:59:34.430062 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.429940 2576 scope.go:117] "RemoveContainer" containerID="7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123" Jul 08 13:59:34.441023 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.441004 2576 scope.go:117] "RemoveContainer" containerID="7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123" Jul 08 13:59:34.441382 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:59:34.441357 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123\": container with ID starting with 7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123 not found: ID does not exist" containerID="7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123" Jul 08 13:59:34.441460 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.441391 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123"} err="failed to get container status \"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123\": rpc error: code = NotFound desc = could not find container \"7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123\": container with ID starting with 7be3ecec7ed2ae9514df2b2e3f12fda045695ea14416dbb5afe5dec326fbd123 not found: ID does not exist" Jul 08 13:59:34.449878 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.449856 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:59:34.457348 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:34.457314 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-4f14d0-694dd8c6c4-bq2c6"] Jul 08 13:59:36.043687 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:36.043650 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93aa133f-357b-4bdc-bca0-8967f9bcec65" path="/var/lib/kubelet/pods/93aa133f-357b-4bdc-bca0-8967f9bcec65/volumes" Jul 08 13:59:40.385765 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.385730 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:59:40.386352 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.385994 2576 kuberuntime_container.go:872] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" podUID="5bcff7d0-dee6-4312-8835-f8fab8989e37" containerName="maas-api" containerID="cri-o://096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8" gracePeriod=30 Jul 08 13:59:40.645379 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.645354 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:59:40.712162 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.712115 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52b6s\" (UniqueName: \"kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s\") pod \"5bcff7d0-dee6-4312-8835-f8fab8989e37\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " Jul 08 13:59:40.712341 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.712166 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls\") pod \"5bcff7d0-dee6-4312-8835-f8fab8989e37\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " Jul 08 13:59:40.712386 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.712356 2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca\") pod \"5bcff7d0-dee6-4312-8835-f8fab8989e37\" (UID: \"5bcff7d0-dee6-4312-8835-f8fab8989e37\") " Jul 08 13:59:40.712690 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.712666 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bcff7d0-dee6-4312-8835-f8fab8989e37" (UID: "5bcff7d0-dee6-4312-8835-f8fab8989e37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 13:59:40.714445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.714417 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bcff7d0-dee6-4312-8835-f8fab8989e37" (UID: "5bcff7d0-dee6-4312-8835-f8fab8989e37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 13:59:40.714445 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.714434 2576 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s" (OuterVolumeSpecName: "kube-api-access-52b6s") pod "5bcff7d0-dee6-4312-8835-f8fab8989e37" (UID: "5bcff7d0-dee6-4312-8835-f8fab8989e37"). InnerVolumeSpecName "kube-api-access-52b6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 13:59:40.813816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.813778 2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bcff7d0-dee6-4312-8835-f8fab8989e37-openshift-service-ca\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:40.813816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.813810 2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52b6s\" (UniqueName: \"kubernetes.io/projected/5bcff7d0-dee6-4312-8835-f8fab8989e37-kube-api-access-52b6s\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:40.813816 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:40.813820 2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bcff7d0-dee6-4312-8835-f8fab8989e37-maas-api-tls\") on node \"ip-10-0-138-212.ec2.internal\" DevicePath \"\"" Jul 08 13:59:41.471406 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.471366 2576 generic.go:356] "Generic (PLEG): container finished" podID="5bcff7d0-dee6-4312-8835-f8fab8989e37" containerID="096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8" exitCode=0 Jul 08 13:59:41.471968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.471415 2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" Jul 08 13:59:41.471968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.471458 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" event={"ID":"5bcff7d0-dee6-4312-8835-f8fab8989e37","Type":"ContainerDied","Data":"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8"} Jul 08 13:59:41.471968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.471528 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw" event={"ID":"5bcff7d0-dee6-4312-8835-f8fab8989e37","Type":"ContainerDied","Data":"6682f3a82aeab02410d051c44fe467f5dcc254e05b57d69a2b5a33a3a01e683a"} Jul 08 13:59:41.471968 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.471565 2576 scope.go:117] "RemoveContainer" containerID="096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8" Jul 08 13:59:41.481215 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.481197 2576 scope.go:117] "RemoveContainer" containerID="096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8" Jul 08 13:59:41.481461 ip-10-0-138-212 kubenswrapper[2576]: E0708 13:59:41.481445 2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8\": container with ID starting with 096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8 not found: ID does not exist" containerID="096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8" Jul 08 13:59:41.481511 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.481469 2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8"} err="failed to get container status \"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8\": rpc error: code = NotFound desc = could not find container \"096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8\": container with ID starting with 096ec8c1590113cc22dc16e2cf117ac3a89467a1a1dbf994439eefba615708f8 not found: ID does not exist" Jul 08 13:59:41.496339 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.496314 2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:59:41.500574 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:41.500552 2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-e225ef-8956b9469-ztwpw"] Jul 08 13:59:42.045090 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:42.045061 2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bcff7d0-dee6-4312-8835-f8fab8989e37" path="/var/lib/kubelet/pods/5bcff7d0-dee6-4312-8835-f8fab8989e37/volumes" Jul 08 13:59:47.193508 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:47.193472 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69899986ff-49d8n_c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e/authorino/0.log" Jul 08 13:59:51.695904 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:51.695869 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cf2hv_e22ac27f-8a4b-4643-bb62-89bb2366dadc/manager/0.log" Jul 08 13:59:51.941388 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:51.941354 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d5d4c5795-gvv6w_734d70ba-2a07-4e45-ab64-289441a400ec/manager/0.log" Jul 08 13:59:52.070078 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:52.069989 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qzhc4_8943a185-42de-4cb3-bff5-d43733e20c47/server/0.log" Jul 08 13:59:52.193087 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:52.193054 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ntkrv_8e35d827-31dc-4c56-bc08-b865ca5337e3/manager/0.log" Jul 08 13:59:52.443623 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:52.443599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774b77d499-88ws8_9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed/copy-manifests/0.log" Jul 08 13:59:52.450827 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:52.450800 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774b77d499-88ws8_9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed/manager/0.log" Jul 08 13:59:53.910481 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:53.910448 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69899986ff-49d8n_c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e/authorino/0.log" Jul 08 13:59:54.032029 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:54.031991 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tzvx6_ba2186af-8af3-45d1-a5f0-2143be5b54ae/manager/0.log" Jul 08 13:59:54.146294 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:54.146263 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-55wfw_8a73aae9-4d19-4b1f-b797-caa2488003d2/manager/0.log" Jul 08 13:59:54.388791 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:54.388763 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zxm65_4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace/registry-server/0.log" Jul 08 13:59:54.757283 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:54.757197 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xfxdg_9e8c574e-e82c-47a0-b0ba-a54de614dfd8/manager/0.log" Jul 08 13:59:55.450208 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:55.450181 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f76655b-jzrxr_1a392e64-8b97-4d85-8327-0d784846b4f7/kube-auth-proxy/0.log" Jul 08 13:59:55.575337 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:55.575300 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/1.log" Jul 08 13:59:55.701708 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:55.701627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-p97vl_44dcf8eb-4c81-48aa-88ce-7ea8b217e018/payload-pre-processing/0.log" Jul 08 13:59:55.821505 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:55.821478 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fs5q7_94864546-69fa-402d-8611-c214e1166ed1/payload-processing/0.log" Jul 08 13:59:55.932930 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:55.932895 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855786878c-jvx86_3a5d7b96-a5a5-469f-99f7-5b5417860995/router/0.log" Jul 08 13:59:56.381922 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:56.381887 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bkjbj_b43a11e4-a795-4354-928a-da90ad4d1f56/main/0.log" Jul 08 13:59:56.610563 ip-10-0-138-212 kubenswrapper[2576]: I0708 13:59:56.610532 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfl8t_bd0f3a20-d752-4cc0-9571-78b559e31669/main/0.log" Jul 08 14:00:01.075296 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.075252 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tljhk/must-gather-w2pmz"] Jul 08 14:00:01.075840 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.075821 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93aa133f-357b-4bdc-bca0-8967f9bcec65" containerName="maas-api" Jul 08 14:00:01.075929 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.075844 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="93aa133f-357b-4bdc-bca0-8967f9bcec65" containerName="maas-api" Jul 08 14:00:01.075929 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.075891 2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bcff7d0-dee6-4312-8835-f8fab8989e37" containerName="maas-api" Jul 08 14:00:01.075929 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.075900 2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bcff7d0-dee6-4312-8835-f8fab8989e37" containerName="maas-api" Jul 08 14:00:01.076079 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.076004 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="93aa133f-357b-4bdc-bca0-8967f9bcec65" containerName="maas-api" Jul 08 14:00:01.076079 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.076016 2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bcff7d0-dee6-4312-8835-f8fab8989e37" containerName="maas-api" Jul 08 14:00:01.079265 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.079243 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.081601 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.081571 2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tljhk\"/\"default-dockercfg-kt4cr\"" Jul 08 14:00:01.081601 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.081595 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tljhk\"/\"openshift-service-ca.crt\"" Jul 08 14:00:01.082154 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.082116 2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tljhk\"/\"kube-root-ca.crt\"" Jul 08 14:00:01.086947 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.086631 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tljhk/must-gather-w2pmz"] Jul 08 14:00:01.186664 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.186629 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlrs\" (UniqueName: \"kubernetes.io/projected/c0653012-d663-4e49-823b-dfec61ce66e3-kube-api-access-mxlrs\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.186852 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.186704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0653012-d663-4e49-823b-dfec61ce66e3-must-gather-output\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.287712 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.287680 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0653012-d663-4e49-823b-dfec61ce66e3-must-gather-output\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.287882 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.287733 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlrs\" (UniqueName: \"kubernetes.io/projected/c0653012-d663-4e49-823b-dfec61ce66e3-kube-api-access-mxlrs\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.288033 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.288011 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c0653012-d663-4e49-823b-dfec61ce66e3-must-gather-output\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.305372 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.305353 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlrs\" (UniqueName: \"kubernetes.io/projected/c0653012-d663-4e49-823b-dfec61ce66e3-kube-api-access-mxlrs\") pod \"must-gather-w2pmz\" (UID: \"c0653012-d663-4e49-823b-dfec61ce66e3\") " pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.390060 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.390029 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tljhk/must-gather-w2pmz" Jul 08 14:00:01.537980 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.537950 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tljhk/must-gather-w2pmz"] Jul 08 14:00:01.539324 ip-10-0-138-212 kubenswrapper[2576]: W0708 14:00:01.539298 2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0653012_d663_4e49_823b_dfec61ce66e3.slice/crio-01f19ac00042c9cd9cfdd902c506ac56b086a6bbfabd6587b9479e44527da6b4 WatchSource:0}: Error finding container 01f19ac00042c9cd9cfdd902c506ac56b086a6bbfabd6587b9479e44527da6b4: Status 404 returned error can't find the container with id 01f19ac00042c9cd9cfdd902c506ac56b086a6bbfabd6587b9479e44527da6b4 Jul 08 14:00:01.575243 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:01.575207 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tljhk/must-gather-w2pmz" event={"ID":"c0653012-d663-4e49-823b-dfec61ce66e3","Type":"ContainerStarted","Data":"01f19ac00042c9cd9cfdd902c506ac56b086a6bbfabd6587b9479e44527da6b4"} Jul 08 14:00:03.590475 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:03.590379 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tljhk/must-gather-w2pmz" event={"ID":"c0653012-d663-4e49-823b-dfec61ce66e3","Type":"ContainerStarted","Data":"dede223ec97399c9d87c6bd592ec0a852b8d35b3e30f7b61e79dcf3e0c32d274"} Jul 08 14:00:03.590475 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:03.590435 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tljhk/must-gather-w2pmz" event={"ID":"c0653012-d663-4e49-823b-dfec61ce66e3","Type":"ContainerStarted","Data":"fa3f88b6ac5fca3a648cbf6c3a1e79ce65df44d8bb3dbd37e501fbeb484b6d29"} Jul 08 14:00:03.606950 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:03.606883 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tljhk/must-gather-w2pmz" podStartSLOduration=0.97440943 podStartE2EDuration="2.606861128s" totalImagesPullingTime="1.632451698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 14:00:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 14:00:03.606113114 +0000 UTC m=+2242.099832905" watchObservedRunningTime="2026-07-08 14:00:03.606861128 +0000 UTC m=+2242.100580918" Jul 08 14:00:04.729226 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:04.729125 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8gljg_c379c49d-db21-4f67-b2a8-b6c871b05a13/global-pull-secret-syncer/0.log" Jul 08 14:00:04.938508 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:04.938475 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z7kws_c6bb7d9e-09e8-4b37-a1eb-a1a6f7a910fa/konnectivity-agent/0.log" Jul 08 14:00:05.008319 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:05.008223 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-212.ec2.internal_bd1788d5073bff9365a683c9a74d77f0/haproxy/0.log" Jul 08 14:00:10.124226 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:10.124187 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-69899986ff-49d8n_c4fb8b4d-b974-4fd0-b80c-4b5eec62f62e/authorino/0.log" Jul 08 14:00:10.157411 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:10.157340 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-tzvx6_ba2186af-8af3-45d1-a5f0-2143be5b54ae/manager/0.log" Jul 08 14:00:10.191194 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:10.190869 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-55wfw_8a73aae9-4d19-4b1f-b797-caa2488003d2/manager/0.log" Jul 08 14:00:10.281654 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:10.281612 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zxm65_4f79b658-c2ba-4d2e-8a47-9ef5a3e4dace/registry-server/0.log" Jul 08 14:00:10.510781 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:10.510750 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-xfxdg_9e8c574e-e82c-47a0-b0ba-a54de614dfd8/manager/0.log" Jul 08 14:00:12.387799 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:12.387761 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-p8wlv_cf4d6258-b250-4459-bc25-e07aa3cbd1f3/cluster-monitoring-operator/0.log" Jul 08 14:00:12.565299 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:12.565266 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z4pm_eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98/node-exporter/0.log" Jul 08 14:00:12.595957 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:12.595923 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z4pm_eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98/kube-rbac-proxy/0.log" Jul 08 14:00:12.620693 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:12.620661 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5z4pm_eecb7c7c-a1a7-4bd9-9d4c-4b2dddf8fa98/init-textfile/0.log" Jul 08 14:00:14.446245 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:14.446206 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-nplcg_ac17ea6d-b1f3-498a-a5d1-ee940a61ec70/networking-console-plugin/0.log" Jul 08 14:00:15.024407 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.024344 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/1.log" Jul 08 14:00:15.033052 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.033023 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-v4bwh_002f971a-3e81-492b-b96e-cc929765f68b/console-operator/2.log" Jul 08 14:00:15.651114 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.651073 2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv"] Jul 08 14:00:15.662324 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.662291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.663735 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.663699 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv"] Jul 08 14:00:15.719486 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.719449 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-nw47v_815297eb-520c-4cc5-aa27-8c53e07d5e27/download-server/0.log" Jul 08 14:00:15.749134 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.749099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-lib-modules\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.749449 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.749405 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-podres\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.749700 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.749683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-sys\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.749903 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.749888 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llp8k\" (UniqueName: \"kubernetes.io/projected/cf2ab440-5811-4add-8d9e-968221b88ca7-kube-api-access-llp8k\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.750089 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.750074 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-proc\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851021 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.850984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-sys\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851239 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851050 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llp8k\" (UniqueName: \"kubernetes.io/projected/cf2ab440-5811-4add-8d9e-968221b88ca7-kube-api-access-llp8k\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851239 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851087 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-proc\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851239 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851121 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-sys\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851239 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851158 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-lib-modules\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851239 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851210 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-podres\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851476 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851211 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-proc\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851476 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-lib-modules\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.851476 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.851367 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cf2ab440-5811-4add-8d9e-968221b88ca7-podres\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.859862 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.859833 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llp8k\" (UniqueName: \"kubernetes.io/projected/cf2ab440-5811-4add-8d9e-968221b88ca7-kube-api-access-llp8k\") pod \"perf-node-gather-daemonset-64vbv\" (UID: \"cf2ab440-5811-4add-8d9e-968221b88ca7\") " pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:15.978165 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:15.978091 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:16.171537 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.169936 2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv"] Jul 08 14:00:16.364406 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.364370 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-gfnd2_33c0f971-0591-4a5a-9f95-acd98429c2ef/volume-data-source-validator/0.log" Jul 08 14:00:16.698822 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.698788 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" event={"ID":"cf2ab440-5811-4add-8d9e-968221b88ca7","Type":"ContainerStarted","Data":"a0c593a0d34232319117613946413737b7bfb8371020fc1246f17226f37bc9db"} Jul 08 14:00:16.699279 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.698846 2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" event={"ID":"cf2ab440-5811-4add-8d9e-968221b88ca7","Type":"ContainerStarted","Data":"e279e71ce1802438d661b07456fe507af1368ef8e72da90045dab19c64e68d68"} Jul 08 14:00:16.699279 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.698870 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:16.720625 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:16.720567 2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" podStartSLOduration=1.720543997 podStartE2EDuration="1.720543997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 14:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 14:00:16.716249973 +0000 UTC m=+2255.209969786" watchObservedRunningTime="2026-07-08 14:00:16.720543997 +0000 UTC m=+2255.214263787" Jul 08 14:00:17.483462 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:17.483433 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnf4_43cc3d2a-0702-464a-a055-88824feaaa42/dns/0.log" Jul 08 14:00:17.502634 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:17.502608 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4wnf4_43cc3d2a-0702-464a-a055-88824feaaa42/kube-rbac-proxy/0.log" Jul 08 14:00:17.568934 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:17.568904 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-cskqh_5058507b-4126-4d96-9618-48ae609b35a3/dns-node-resolver/0.log" Jul 08 14:00:18.230062 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:18.230033 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-sgrrx_9508464b-6316-489b-a555-5eb6b3304982/node-ca/0.log" Jul 08 14:00:19.332293 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.332259 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f76655b-jzrxr_1a392e64-8b97-4d85-8327-0d784846b4f7/kube-auth-proxy/0.log" Jul 08 14:00:19.368102 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.368069 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/0.log" Jul 08 14:00:19.426370 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.426333 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/1.log" Jul 08 14:00:19.482879 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.482849 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-p97vl_44dcf8eb-4c81-48aa-88ce-7ea8b217e018/payload-pre-processing/0.log" Jul 08 14:00:19.556873 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.556838 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fs5q7_94864546-69fa-402d-8611-c214e1166ed1/payload-processing/0.log" Jul 08 14:00:19.579045 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:19.579005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855786878c-jvx86_3a5d7b96-a5a5-469f-99f7-5b5417860995/router/0.log" Jul 08 14:00:20.284495 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:20.284461 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rsqrk_6813cb3b-ee4c-405a-b290-7845d985063b/serve-healthcheck-canary/0.log" Jul 08 14:00:20.793684 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:20.793656 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-vkv4v_1d17a6d5-89cb-4b70-a32a-584158100116/insights-operator/0.log" Jul 08 14:00:20.882394 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:20.882358 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk28s_1353abf6-046c-4345-b792-90b2da7941a7/kube-rbac-proxy/0.log" Jul 08 14:00:20.906712 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:20.906682 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk28s_1353abf6-046c-4345-b792-90b2da7941a7/exporter/0.log" Jul 08 14:00:20.928697 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:20.928669 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jk28s_1353abf6-046c-4345-b792-90b2da7941a7/extractor/0.log" Jul 08 14:00:22.719002 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:22.718970 2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tljhk/perf-node-gather-daemonset-64vbv" Jul 08 14:00:23.270828 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.270786 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cf2hv_e22ac27f-8a4b-4643-bb62-89bb2366dadc/manager/0.log" Jul 08 14:00:23.519844 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.519813 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d5d4c5795-gvv6w_734d70ba-2a07-4e45-ab64-289441a400ec/manager/0.log" Jul 08 14:00:23.539094 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.539020 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qzhc4_8943a185-42de-4cb3-bff5-d43733e20c47/server/0.log" Jul 08 14:00:23.646126 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.646094 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-ntkrv_8e35d827-31dc-4c56-bc08-b865ca5337e3/manager/0.log" Jul 08 14:00:23.762713 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.762676 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774b77d499-88ws8_9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed/manager/0.log" Jul 08 14:00:23.783651 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:23.783625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-774b77d499-88ws8_9e6e11db-ecc3-4160-bda7-ea2c3f4dd2ed/copy-manifests/0.log" Jul 08 14:00:25.694282 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.694251 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f76655b-jzrxr_1a392e64-8b97-4d85-8327-0d784846b4f7/kube-auth-proxy/0.log" Jul 08 14:00:25.716285 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.716248 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/0.log" Jul 08 14:00:25.774256 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.774221 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/1.log" Jul 08 14:00:25.827996 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.827968 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-p97vl_44dcf8eb-4c81-48aa-88ce-7ea8b217e018/payload-pre-processing/0.log" Jul 08 14:00:25.910850 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.910814 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fs5q7_94864546-69fa-402d-8611-c214e1166ed1/payload-processing/0.log" Jul 08 14:00:25.928302 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:25.928264 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855786878c-jvx86_3a5d7b96-a5a5-469f-99f7-5b5417860995/router/0.log" Jul 08 14:00:27.211597 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.211529 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746f76655b-jzrxr_1a392e64-8b97-4d85-8327-0d784846b4f7/kube-auth-proxy/0.log" Jul 08 14:00:27.234627 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.234599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/0.log" Jul 08 14:00:27.292230 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.292199 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-fn5sw_30918a42-4d90-4bff-83ce-b2facafdd2b8/istio-proxy/1.log" Jul 08 14:00:27.340429 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.340393 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-p97vl_44dcf8eb-4c81-48aa-88ce-7ea8b217e018/payload-pre-processing/0.log" Jul 08 14:00:27.404195 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.402644 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-fs5q7_94864546-69fa-402d-8611-c214e1166ed1/payload-processing/0.log" Jul 08 14:00:27.427629 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:27.427599 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-855786878c-jvx86_3a5d7b96-a5a5-469f-99f7-5b5417860995/router/0.log" Jul 08 14:00:30.351137 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:30.351082 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-w5h89_f112cec5-a7e4-42d2-9c08-34c0dca62bef/migrator/0.log" Jul 08 14:00:30.384772 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:30.384738 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-w5h89_f112cec5-a7e4-42d2-9c08-34c0dca62bef/graceful-termination/0.log" Jul 08 14:00:30.782286 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:30.782250 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-l7dm8_a9e657f5-aaf8-4a08-a567-2346b505dfd7/kube-storage-version-migrator-operator/1.log" Jul 08 14:00:30.783654 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:30.783627 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-l7dm8_a9e657f5-aaf8-4a08-a567-2346b505dfd7/kube-storage-version-migrator-operator/0.log" Jul 08 14:00:31.850221 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:31.850158 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-7f84b_0ae0a921-745f-411e-8ec7-33fcfb7a5bd3/kube-multus/0.log" Jul 08 14:00:31.915119 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:31.915092 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/kube-multus-additional-cni-plugins/0.log" Jul 08 14:00:31.939648 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:31.939625 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/egress-router-binary-copy/0.log" Jul 08 14:00:31.967321 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:31.967296 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/cni-plugins/0.log" Jul 08 14:00:31.990493 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:31.990469 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/bond-cni-plugin/0.log" Jul 08 14:00:32.010845 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:32.010814 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/routeoverride-cni/0.log" Jul 08 14:00:32.030026 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:32.030005 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/whereabouts-cni-bincopy/0.log" Jul 08 14:00:32.049832 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:32.049807 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-q6csc_597e86f8-e61a-4172-a71b-77e69cbd0fe5/whereabouts-cni/0.log" Jul 08 14:00:32.457636 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:32.457610 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pjblf_a1804049-94ed-4c4c-a122-c8ba0b2ca304/network-metrics-daemon/0.log" Jul 08 14:00:32.474022 ip-10-0-138-212 kubenswrapper[2576]: I0708 14:00:32.473964 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-pjblf_a1804049-94ed-4c4c-a122-c8ba0b2ca304/kube-rbac-proxy/0.log"